Runtime Optimization

Optimize inference throughput and latency: CUDA Graphs for kernel-replay, pre-allocated output buffers, and the Python runtime module.

• CUDAGraphs and the Output Allocator
• Example: Torch Export with Cudagraphs
• Example: Pre-allocated output buffer
• Python Runtime