Runtime Optimization

Optimize inference throughput and latency: CUDA Graphs for kernel-replay, pre-allocated output buffers, and choosing the Python vs C++ TRT execution path.

• CUDAGraphs and the Output Allocator
• Example: Torch Export with Cudagraphs
• Example: Pre-allocated output buffer
• Python vs C++ runtime