Advanced Usage

Step-by-step tutorials covering engine caching, quantization, custom kernels, dynamic shapes, weight streaming, debugging, and more.

• HuggingFace Models
  • Compiling LLM models from Huggingface
  • Example: Compiling Stable Diffusion with torch.compile
  • Example: Compiling FLUX.1-dev with the dynamo backend
  • Example: Mutable Torch TensorRT Module
• Extensibility
  • Converters
  • Lowering Passes
  • Plugins
• Resource & Memory Management
  • Resource Management
  • Engine Caching
  • Example: Engine Caching
  • Example: Engine Caching (BERT)
  • Example: Weight Streaming
  • Example: Dynamic Memory Allocation
  • Example: Low CPU Memory Compilation
• Compilation & Graph Analysis
  • Tracing Models with torch_tensorrt.dynamo.trace
  • Dryrun Mode
  • Example: Hierarchical Partitioner
• Weight Refitting & LoRA
  • Refitting TensorRT Engines with Updated Weights
  • Example: Refitting Programs with New Weights
• Runtime Optimization
  • CUDAGraphs and the Output Allocator
  • Example: Torch Export with Cudagraphs
  • Example: Pre-allocated output buffer
  • Python Runtime
• Deployment
  • Serving a Torch-TensorRT model with Triton
  • Cross-Compiling for Windows
  • Example: Cross-runtime Compilation for Windows
  • Distributed Inference
  • Complex Tensor Support
• Example: Distributed Inference
• Operators Supported
  • ATen Core Ops — Converted
  • ATen Core Ops — Lowered via Decomposition
  • Python Builtin Ops — Converted
  • Prims Ops — Converted
  • Prims Ops — Lowered via Decomposition