torch_tensorrt.dynamo#

Functions#

torch_tensorrt.dynamo.compile(exported_program: ExportedProgram, inputs: Sequence[Sequence[Any]] | None = None, *, arg_inputs: Sequence[Sequence[Any]] | None = None, kwarg_inputs: dict[Any, Any] | None = None, device: Device | device | str | None = None, disable_tf32: bool = False, assume_dynamic_shape_support: bool = False, sparse_weights: bool = False, engine_capability: EngineCapability = EngineCapability.STANDARD, num_avg_timing_iters: int = 1, workspace_size: int = 0, dla_sram_size: int = 1048576, dla_local_dram_size: int = 1073741824, dla_global_dram_size: int = 536870912, truncate_double: bool = False, use_complex_decomposition: bool = True, require_full_compilation: bool = False, min_block_size: int = 5, torch_executed_ops: Collection[Callable[[...], Any] | str] | None = None, torch_executed_modules: Collection[str] | None = None, pass_through_build_failures: bool = False, max_aux_streams: int | None = None, version_compatible: bool = False, optimization_level: int | None = None, use_python_runtime: bool = False, use_fast_partitioner: bool = True, enable_experimental_decompositions: bool = False, dryrun: bool = False, hardware_compatible: bool = False, target_compute_capabilities: List[Tuple[int, int]] | None = None, timing_cache_path: str = '/tmp/torch_tensorrt_engine_cache/timing_cache.bin', lazy_engine_init: bool = False, cache_built_engines: bool = False, reuse_cached_engines: bool = False, engine_cache_dir: str = '/tmp/torch_tensorrt_engine_cache', engine_cache_size: int = 5368709120, custom_engine_cache: BaseEngineCache | None = None, use_fp32_acc: bool = False, refit_identical_engine_weights: bool = False, strip_engine_weights: bool = False, immutable_weights: bool = True, enable_weight_streaming: bool = False, tiling_optimization_level: str = 'none', l2_limit_for_tiling: int = -1, offload_module_to_cpu: bool = False, use_distributed_mode_trace: bool = False, enable_autocast: bool = False, autocast_low_precision_type: dtype | dtype | None = None, autocast_excluded_nodes: Collection[str] = {}, autocast_excluded_node_names: Collection[str] = {}, autocast_excluded_ops: Collection[Callable[[...], Any] | str] = {}, autocast_max_output_threshold: float = 512, autocast_max_depth_of_reduction: int | None = None, autocast_calibration_dataloader: DataLoader | None = None, autocast_model_regions: str = 'keep', cpu_memory_budget: int | None = None, max_partition_size: int | None = None, enable_resource_partitioning: bool = False, dynamically_allocate_resources: bool = False, release_weights_after_build: bool | None = None, decompose_attention: bool = False, disabled_constant_fold_exclusions: Collection[str] = frozenset({}), attn_bias_is_causal: bool | str = True, fold_shared_kv_attention: bool = False, deduplicate_weights: bool = False, fallback_data_dependent_ops: bool = False, fallback_pytorch_plugin_ops: bool = False, collapse_high_rank_permutes: bool = False, convert_antialiased_resize: bool = False, replace_interleave_stack_with_cat: bool = False, build_route: str | None = None, tune_build_routes: str | None = None, tune_build_route_file: str | None = None, tuning_search: str = 'fast', tuning_timeout_s: int = -1, tuning_cache_file: str | None = None, tuning_continue: bool = False, tuning_dry_run: bool = False, accuracy_threshold: float | None = None, accuracy_algorithm: str = 'l0', accuracy_atol: float = 1e-05, accuracy_rtol: float = 1e-05, **kwargs: Any) → GraphModule[source]#

Compile an ExportedProgram module for NVIDIA GPUs using TensorRT

Takes a existing TorchScript module and a set of settings to configure the compiler and will convert methods to JIT Graphs which call equivalent TensorRT engines

Converts specifically the forward method of a TorchScript Module

Parameters:
  • exported_program (torch.export.ExportedProgram) – Source module, running torch.export on a torch.nn.Module

  • inputs (Optional[Sequence[Sequence[Any]]]) –

    List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type.

    inputs=[
        torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1
        torch_tensorrt.Input(
            min_shape=(1, 224, 224, 3),
            opt_shape=(1, 512, 512, 3),
            max_shape=(1, 1024, 1024, 3),
            dtype=torch.int32
            format=torch.channel_last
        ), # Dynamic input shape for input #2
        torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings
    ]
    

Keyword Arguments:
  • arg_inputs (Optional[Sequence[Sequence[Any]]]) – Same as inputs. Alias for better understanding with kwarg_inputs.

  • kwarg_inputs (Optional[dict[Any, Any]]) – kwarg inputs to the module forward function.

  • device (Union(Device, torch.device, dict)) –

    Target device for TensorRT engines to run on

    device=torch_tensorrt.Device("dla:1", allow_gpu_fallback=True)
    

  • disable_tf32 (bool) – Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas

  • assume_dynamic_shape_support (bool) – Setting this to true enables the converters work for both dynamic and static shapes. Default: False

  • sparse_weights (bool) – Enable sparsity for convolution and fully connected layers.

  • engine_capability (EngineCapability) – Restrict kernel selection to safe gpu kernels or safe dla kernels

  • num_avg_timing_iters (python:int) – Number of averaging timing iterations used to select kernels

  • workspace_size (python:int) – Maximum size of workspace given to TensorRT

  • dla_sram_size (python:int) – Fast software managed RAM used by DLA to communicate within a layer.

  • dla_local_dram_size (python:int) – Host RAM used by DLA to share intermediate tensor data across operations

  • dla_global_dram_size (python:int) – Host RAM used by DLA to store weights and metadata for execution

  • truncate_double (bool) – Truncate weights provided in double (float64) to float32

  • use_complex_decomposition (bool) – Use PyTorch’s upstream complex decomposition (requires torch>=2.14) instead of the legacy hand-rolled complex rewriter. Falls back to the legacy pass when unavailable.

  • require_full_compilation (bool) – Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch

  • min_block_size (python:int) – The minimum number of contiguous TensorRT convertible operations in order to run a set of operations in TensorRT

  • torch_executed_ops (Optional[Collection[Target]]) – Set of aten operators that must be run in PyTorch. An error will be thrown if this set is not empty but require_full_compilation is True

  • torch_executed_modules (Optional[Collection[str]]) – Collection of modules that must be run in PyTorch. An error will be thrown if this collection is not empty but require_full_compilation is True

  • pass_through_build_failures (bool) – Error out if there are issues during compilation (only applicable to torch.compile workflows)

  • max_aux_streams (Optional[python:int]) – Maximum streams in the engine

  • version_compatible (bool) – Build the TensorRT engines compatible with future versions of TensorRT (Restrict to lean runtime operators to provide version forward compatibility for the engines)

  • optimization_level – (Optional[int]): Setting a higher optimization level allows TensorRT to spend longer engine building time searching for more optimization options. The resulting engine may have better performance compared to an engine built with a lower optimization level. The default optimization level is 3. Valid values include integers from 0 to the maximum optimization level, which is currently 5. Setting it to be greater than the maximum level results in identical behavior to the maximum level.

  • use_python_runtime – (bool): Deprecated. Kept for backward compatibility; emits a DeprecationWarning when set to True. The Python and C++ runtimes are now merged and the runtime is selected automatically based on whether the C++ Torch-TensorRT runtime is available.

  • use_fast_partitioner – (bool): Use the adjacency based partitioning scheme instead of the global partitioner. Adjacency partitioning is faster but may not be optimal. Use the global paritioner (False) if looking for best performance

  • enable_experimental_decompositions (bool) – Use the full set of operator decompositions. These decompositions may not be tested but serve to make the graph easier to convert to TensorRT, potentially increasing the amount of graphs run in TensorRT.

  • dryrun (bool) – Toggle for “Dryrun” mode, running everything except conversion to TRT and logging outputs

  • hardware_compatible (bool) – Build the TensorRT engines compatible with GPU architectures other than that of the GPU on which the engine was built (currently works for NVIDIA Ampere and newer)

  • target_compute_capabilities (Optional[List[Tuple[python:int, python:int]]]) – Compute capabilities to build for, e.g. [(7, 5)] for Turing. Defaults to None. The compilation then targets the current device, from torch.cuda.get_device_capability(). TensorRT-RTX only. Drives both engine targeting and op partitioning, so ops unsupported on any listed target fall back to PyTorch.

  • timing_cache_path (str) – Path to the timing cache if it exists (or) where it will be saved after compilation. Not used for TensorRT-RTX.

  • lazy_engine_init (bool) – Defer setting up engines until the compilation of all engines is complete. Can allow larger models with multiple graph breaks to compile but can lead to oversubscription of GPU memory at runtime.

  • cache_built_engines (bool) – Whether to save the compiled TRT engines to storage

  • reuse_cached_engines (bool) – Whether to load the compiled TRT engines from storage

  • engine_cache_dir (str) – Directory to store the cached TRT engines

  • engine_cache_size (python:int) – Maximum hard-disk space (bytes) to use for the engine cache, default is 1GB. If the cache exceeds this size, the oldest engines will be removed by default

  • custom_engine_cache (Optional[BaseEngineCache]) – Engine cache instance to use for saving and loading engines. Users can provide their own engine cache by inheriting from BaseEngineCache. If used, engine_cache_dir and engine_cache_size will be ignored.

  • use_fp32_acc (bool) – Enable FP32 accumulation for FP16 matmul layers while retaining FP16 inputs and outputs. When combined with decompose_attention=True, the complete decomposed FP16 scaled dot product attention calculation runs in FP32 and only its final output is cast back to FP16. This option has no effect on FP32 or BF16 inputs.

  • refit_identical_engine_weights (bool) – Refit engines with identical weights. This is useful when the same model is compiled multiple times with different inputs and the weights are the same. This will save time by reusing the same engine for different inputs.

  • strip_engine_weights (bool) – Strip engine weights from the serialized engine. This is useful when the engine is to be deployed in an environment where the weights are not required.

  • immutable_weights (bool) – Build non-refittable engines. This is useful for some layers that are not refittable. If this argument is set to true, strip_engine_weights and refit_identical_engine_weights will be ignored.

  • enable_weight_streaming (bool) – Enable weight streaming.

  • tiling_optimization_level (str) – The optimization level of tiling strategies. A higher level allows TensorRT to spend more time searching for better tiling strategy. We currently support [“none”, “fast”, “moderate”, “full”].

  • l2_limit_for_tiling (python:int) – The target L2 cache usage limit (in bytes) for tiling optimization (default is -1 which means no limit).

  • offload_module_to_cpu (bool) – Offload the module to CPU. This is useful when we need to minimize GPU memory usage.

  • use_distributed_mode_trace (bool) – Using aot_autograd to trace the graph. This is enabled when DTensors or distributed tensors are present in distributed model

  • enable_autocast (bool) – Whether to enable autocast.

  • autocast_low_precision_type (Optional[Union[torch.dpython:type, dpython:type]]) – The precision to reduce to. We currently support torch.float16 and torch.bfloat16. Default is None, which means no low precision is used.

  • autocast_excluded_nodes (Collection[str]) – The set of regex patterns for the paths of the modules, such as “block.norm”, whose nodes should remain in FP32. Default is [].

  • autocast_excluded_node_names (Collection[str]) – The set of regex patterns for the names of the nodes in the exported graph, such as “layer_norm_1”, that should remain in FP32. A pattern matches from the start of a name, so “layer_norm_1$” matches layer_norm_1 and not layer_norm_10. Default is [].

  • autocast_excluded_ops (Collection[Target]) – The set of targets (ATen ops) that should remain in FP32. Default is [].

  • autocast_max_output_threshold (python:float) – Maximum absolute value for node outputs, nodes with outputs greater than this value will remain in FP32. Needs autocast_calibration_dataloader. Default is 512.

  • autocast_max_depth_of_reduction (Optional[python:int]) – Maximum depth of reduction allowed in low precision. Nodes with higher reduction depths will remain in FP32. This helps prevent excessive accuracy loss in operations particularly sensitive to reduced precision, as higher-depth reductions may amplify computation errors in low precision formats. If not provided, infinity will be used. Needs autocast_calibration_dataloader. Default is None.

  • autocast_calibration_dataloader (Optional[torch.utils.data.DataLoader]) – The dataloader to use for autocast calibration. Default is None.

  • autocast_model_regions (str) – What autocast does with the regions the model itself runs under torch.autocast. “keep” builds them with the precisions that PyTorch autocast gives them. “reclassify” removes their autocast, so that the autocast rules, exclusions and calibration of Torch-TensorRT choose the precision of every op in them too. Regions with enabled=False, which a model uses to keep ops in FP32, keep their autocast. So do regions whose values inlining would change, such as one that writes into a buffer in place, and regions inside a torch.no_grad() block when the model is exported with grad mode on. autocast_excluded_node_names also matches the names that ops have in the regions. Default is “keep”.

  • enable_resource_partitioning (bool) – Enable resource-aware partitioning. This is useful when the model is large and the CPU memory is limited.

  • cpu_memory_budget (Optional[python:int]) – With resource partitioning, the host memory in bytes the compilation may use: each TensorRT engine is sized to a quarter of what the process’s resident memory leaves of it. Defaults to the available system memory.

  • max_partition_size (Optional[python:int]) – The weight bytes per TensorRT engine the resource partitioner splits to: a finer control than cpu_memory_budget, which it replaces, so the two can’t both be set. The size is then not derived from the budget and the process’s resident memory, so the same model and size always give the same split. Setting it turns resource partitioning on.

  • dynamically_allocate_resources (bool) – Dynamically allocate resources during engine execution.

  • release_weights_after_build (Optional[bool]) – Release the memory of each model weight once the last TensorRT engine that reads it is built, and of weights no partition reads before the first build, so compilation holds only the weights still to be built. None (default) releases only weights mapped by map_weights: their pages are dropped and read back if used again, so the source module keeps working. True also frees ordinary weights; the source module, and any tensor sharing storage with its weights, must not be used afterwards, even if compilation fails. False releases nothing. Weights read by subgraphs that run in PyTorch are kept. Applies to torch_tensorrt.dynamo.compile; True is turned off, with a warning, under torch.compile. MutableTorchTensorRTModule uses False.

  • decompose_attention (bool) – Whether to decompose attention layers into smaller operations instead of using the attention converters. When combined with use_fp32_acc=True, decomposed FP16 attention keeps its intermediate calculation in FP32 and casts only the final output back to FP16.

  • disabled_constant_fold_exclusions (Collection[str]) – IDs of predefined Torch-TensorRT rules to disable. Rules are registered by lowering implementations and enabled by default; compilation users do not need to register them. Disabling a rule makes the nodes that only it matches foldable. Supported IDs: "attention_mask_arange", "scatter_add_positions". Default is empty.

  • attn_bias_is_causal (Union[bool, str]) – Which attn_bias of efficient SDPA to replace with TensorRT’s built-in causal mask, which is faster and more accurate. True: a bias proven causal, one that depends on no graph input and holds 0 on and below the diagonal and -inf (or the dtype’s lowest value) above, and also a bias that cannot be proven, such as one built from an input, with a dynamic sequence length, or from a module buffer under torch.compile; you assert that it is causal. “proven”: only a bias proven causal, for models whose input-built masks may not be causal, such as masks with padding. A bias that is computed and found not causal is always kept. A mask longer than 4096, or one whose check fails, is not proven: True replaces it and “proven” keeps it. False: never replace. A replaced bias is not read again, so refitting the weights it came from does not change the engine. Default is True.

  • fold_shared_kv_attention (bool) – For attention where all query heads share one key/value head (multi-query attention), compute it against that one head instead of the copies that Hugging Face models make for each query head. TensorRT then runs two plain matrix products instead of its fused attention kernel, which can be faster or slower depending on the GPU, so measure it. In float16 the result can round differently; use_fp32_acc=True reduces that. Only static shapes are rewritten, and scaled_dot_product_attention also needs decompose_attention=True. Both preserved matmuls and the reshape/bmm form produced by AOT tracing under torch.compile are supported for repeat_kv style graphs. In the bmm form, broadcasting a single KV head with enable_gqa=True is not folded. Default is False.

  • deduplicate_weights (bool) – Add a weight that several layers read to the TensorRT network once instead of once per layer, and give the linear layers that share a weight one transposed copy of it. This cuts engine-build host memory for models that reuse weights, such as a denoising loop unrolled into one graph, most of all in float32 builds with disable_tf32=True and in refittable engines. TensorRT may pick other kernels, so results can round differently, and some models need more memory instead, so measure it. Default is False.

  • fallback_data_dependent_ops (bool) – If True, operators whose converters require a TensorRT output allocator (i.e. data-dependent output shapes, such as nonzero) are added to torch_executed_ops and run in PyTorch instead of being lowered into a TensorRT engine. This is useful when targeting runtimes that cannot consume a TensorRT output allocator. Default is False.

  • fallback_pytorch_plugin_ops (bool) – If True, operators whose converters insert a TensorRT plugin that runs PyTorch (such as aten.index_put with accumulate=True, which uses Torch-TensorRT’s ScatterAdd plugin, or a generated plugin without an AOT kernel) run in PyTorch instead of being lowered into a TensorRT engine. An accumulating aten.index_put with one 1-D index then stays in TensorRT through TensorRT’s ScatterElements plugin, which needs no PyTorch. This needs float32, float16, bfloat16, int32 or int64 data. The input and the values it receives (the input’s shape with the indexed dimension set to the index length) must stay under 2**31 elements, so every dynamic size needs an upper bound in the exported program, such as torch.export.Dim(max=…), that keeps them there. Otherwise the operator runs in PyTorch. Use it for runtimes without PyTorch, such as the ExecuTorch TensorRT runtime. Default is False.

  • collapse_high_rank_permutes (bool) – Rewrite a reshape, permute, reshape chain over more than 8 dimensions, TensorRT’s limit, as reshapes and permutes of at most 8 dimensions that give the same values. Window and patch layouts, such as neighborhood attention and the Qwen2-VL image patches, build such chains. Left in PyTorch, each one splits the graph, and the partitioner can then hand many large tensors across one engine boundary. Only static shapes are rewritten. Fewer, larger engines can need much more time and memory to build and to run, so measure it. Default is False.

  • convert_antialiased_resize (bool) – Run antialiased bilinear and bicubic resize (F.interpolate with antialias=True) in TensorRT instead of PyTorch, so it does not split the graph. Only static input and output heights and widths and floating-point inputs are converted. Some sizes run slower than in PyTorch, so measure it. Default is False.

  • replace_interleave_stack_with_cat (bool) – Write a stack whose only user is a reshape that merges the new axis into a neighbor axis, as in torch.stack([even, odd], dim=3).flatten(2, 3) for a 2x upsample, as a concatenation of unsqueezed inputs. This can make large, image-sized interleaves faster on GPUs where TensorRT fuses the region around them, but slower where TensorRT runs them as separate layers and adds copies, so measure it. Only programs from torch.export are rewritten, not torch.compile graphs. Default is False.

  • build_route (Optional[str]) – TensorRT Global Performance Tuning build route string (space-separated “-knob=value” tokens). None uses the default route. This feature requires TensorRT with Global Performance Tuning enabled; currently unavailable on TensorRT-RTX / Windows.

  • tune_build_routes (Optional[str]) – Build-route expression for an autotuning sweep that runs each candidate in a spawned child process (e.g. “-slice_fusion=[on|off] -kgen:codegen:cuda_tile=[0|1|2|3]”). None disables tuning.

  • tune_build_route_file (Optional[str]) – Path to a file with one route token per line (same as “tune_build_routes”). Mutually exclusive with “tune_build_routes”.

  • tuning_search (str) – Search algorithm: “fast”, “full”, or “mixed”. Default is “fast”. “fast” runs a baseline with all knobs at default, then varies one knob at a time. “full” runs a full grid search. “mixed” runs “fast” first, and then “full” only on knobs that improved performance.

  • tuning_timeout_s (python:int) – Time budget for the entire tuning process. The current iteration finishes before the loop stops. Use -1 (default) to disable the timeout. Helpful for capping large “full” sweeps.

  • tuning_cache_file (Optional[str]) – JSONL path for tuning results / resume.

  • tuning_continue (bool) – Resume an interrupted sweep from “tuning_cache_file”. When it is True, “tuning_cache_file” must be provided; “tune_build_routes”, “tune_build_route_file”, and “tuning_dry_run” must not be provided.

  • tuning_dry_run (bool) – Enumerate candidate routes without building engines.

  • accuracy_threshold (Optional[python:float]) – Max allowed accuracy loss between the engine outputs and eager Torch reference outputs; routes with accuracy loss above this threshold are excluded from best-engine selection. “None” skips accuracy checks.

  • accuracy_algorithm (str) – Loss metric: “l0”, “l1”, “l2”, “lInf”, or “cos” (lower is better).

  • accuracy_atol (python:float) – Absolute tolerance for “accuracy_algorithm=”l0””.

  • accuracy_rtol (python:float) – Relative tolerance for “accuracy_algorithm=”l0””.

  • **kwargs – Any,

Returns:

Compiled FX Module, when run it will execute via TensorRT

Return type:

torch.fx.GraphModule

torch_tensorrt.dynamo.trace(mod: Module | GraphModule, inputs: Tuple[Any, ...] | None = None, *, arg_inputs: Tuple[Any, ...] | None = None, kwarg_inputs: dict[Any, Any] | None = None, **kwargs: Any) → ExportedProgram[source]#

Exports a torch.export.ExportedProgram from a torch.nn.Module or torch.fx.GraphModule specifically targeting being compiled with Torch-TensorRT

Exports a torch.export.ExportedProgram from either a torch.nn.Module or torch.fx.GraphModule``. Runs specific operator decompositions geared towards compilation by Torch-TensorRT’s dynamo frontend.

Parameters:
  • mod (torch.nn.Module | torch.fx.GraphModule) – Source module to later be compiled by Torch-TensorRT’s dynamo fronted

  • inputs (Tuple[Any, ...]) –

    List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type.

    input=[
        torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1
        torch_tensorrt.Input(
            min_shape=(1, 224, 224, 3),
            opt_shape=(1, 512, 512, 3),
            max_shape=(1, 1024, 1024, 3),
            dtype=torch.int32
            format=torch.channel_last
        ), # Dynamic input shape for input #2
        torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings
    ]
    

Keyword Arguments:
  • arg_inputs (Tuple[Any, ...]) – Same as inputs. Alias for better understanding with kwarg_inputs.

  • kwarg_inputs (dict[Any, ...]) – Optional, kwarg inputs to the module forward function.

  • device (Union(torch.device, dict)) –

    Target device for TensorRT engines to run on

    device=torch.device("cuda:0")
    

  • debug (bool) – Enable debuggable engine

  • enable_experimental_decompositions (bool) – Use the full set of operator decompositions. These decompositions may not be tested but serve to make the graph easier to convert to TensorRT, potentially increasing the amount of graphs run in TensorRT.

  • **kwargs – Any,

Returns:

Compiled FX Module, when run it will execute via TensorRT

Return type:

torch.fx.GraphModule

torch_tensorrt.dynamo.export(gm: GraphModule, *, arg_inputs: Sequence[Tensor] | None = None, kwarg_inputs: Dict[str, Any] | None = None, dynamic_shapes: Dict[str, Any] | None = None, cross_compile_module: bool | None = False, use_legacy_exporter: bool | None = False, share_tensors: bool = False) → ExportedProgram[source]#

Export the result of TensorRT compilation into the desired output format.

Parameters:
  • gm (torch.fx.GraphModule) – Compiled Torch-TensorRT module, generated by torch_tensorrt.dynamo.compile

  • inputs (torch.Tensor) – Torch input tensors

  • cross_compile_module (bool) – Flag to indicated whether it is cross_compilation enabled or not

  • share_tensors (bool) – Give the program gm’s parameters, buffers and constants instead of copies of them, which saves holding every weight twice. Only for a program that is read and then discarded: a write into the program’s tensors, or a backend that frees them after lowering, changes gm too, and an in-place refit of gm changes the program’s PyTorch weights but not its engines.

torch_tensorrt.dynamo.convert_exported_program_to_serialized_trt_engine(exported_program: ExportedProgram, inputs: Sequence[Sequence[Any]] | None = None, *, arg_inputs: Sequence[Sequence[Any]] | None = None, kwarg_inputs: dict[Any, Any] | None = None, device: Device | device | str | None = None, disable_tf32: bool = False, assume_dynamic_shape_support: bool = False, sparse_weights: bool = False, engine_capability: EngineCapability = EngineCapability.STANDARD, num_avg_timing_iters: int = 1, workspace_size: int = 0, dla_sram_size: int = 1048576, dla_local_dram_size: int = 1073741824, dla_global_dram_size: int = 536870912, truncate_double: bool = False, use_complex_decomposition: bool = True, require_full_compilation: bool = False, min_block_size: int = 5, torch_executed_ops: Collection[Callable[[...], Any] | str] | None = None, torch_executed_modules: Collection[str] | None = None, pass_through_build_failures: bool = False, max_aux_streams: int | None = None, version_compatible: bool = False, optimization_level: int | None = None, use_python_runtime: bool = False, use_fast_partitioner: bool = True, enable_experimental_decompositions: bool = False, dryrun: bool = False, hardware_compatible: bool = False, target_compute_capabilities: List[Tuple[int, int]] | None = None, timing_cache_path: str = '/tmp/torch_tensorrt_engine_cache/timing_cache.bin', lazy_engine_init: bool = False, cache_built_engines: bool = False, reuse_cached_engines: bool = False, engine_cache_dir: str = '/tmp/torch_tensorrt_engine_cache', engine_cache_size: int = 5368709120, custom_engine_cache: BaseEngineCache | None = None, use_fp32_acc: bool = False, refit_identical_engine_weights: bool = False, strip_engine_weights: bool = False, immutable_weights: bool = True, enable_weight_streaming: bool = False, tiling_optimization_level: str = 'none', l2_limit_for_tiling: int = -1, offload_module_to_cpu: bool = False, use_distributed_mode_trace: bool = False, decompose_attention: bool = False, disabled_constant_fold_exclusions: Collection[str] = frozenset({}), attn_bias_is_causal: bool | str = True, fold_shared_kv_attention: bool = False, deduplicate_weights: bool = False, fallback_pytorch_plugin_ops: bool = False, collapse_high_rank_permutes: bool = False, convert_antialiased_resize: bool = False, replace_interleave_stack_with_cat: bool = False, lift_mutable_buffers: bool = False, arg_input_binding_names: Any = None, kwarg_input_binding_names: Any = None, output_binding_names: Any = None, **kwargs: Any) → bytes[source]#

Convert an ExportedProgram to a serialized TensorRT engine

Converts an ExportedProgram to a serialized TensorRT engine given a dictionary of conversion settings

When lift_mutable_buffers=True, any module buffer that the model mutates (a BUFFER_MUTATION in the EP’s graph signature) is lifted from a baked-in constant to an engine input binding. The resulting engine has additional input bindings appended after the user-supplied inputs, in the order the buffers appear in the EP. The caller is responsible for threading those bindings at runtime — pass the current buffer values in on each call; the engine writes through the binding via aliased I/O so the buffer’s storage is mutated in place. Use trt.ICudaEngine.get_aliased_input_tensor (or the metadata exposed by TRTEngine via aliased_io) to discover which output binding aliases which input. The higher-level torch_tensorrt.dynamo.compile() does this lifting and threading automatically; this lower-level entry point exposes the same machinery for callers that want to manage the bindings themselves.

Only writes the engine can alias in place are supported here. A mutation the engine cannot alias needs its new value copied back into the buffer after the call, and this entry point reports neither which output carries which buffer nor performs the copy, so it raises rather than returning an engine whose buffer would never update. Use torch_tensorrt.dynamo.compile() for those models.

Parameters:
  • exported_program (torch.export.ExportedProgram) – Source module, running torch.export on a torch.nn.Module

  • inputs (Optional[Sequence[Sequence[Any]]]) –

    List of specifications of input shape, dtype and memory layout for inputs to the module. This argument is required. Input Sizes can be specified as torch sizes, tuples or lists. dtypes can be specified using torch datatypes or torch_tensorrt datatypes and you can use either torch devices or the torch_tensorrt device type enum to select device type.

    inputs=[
        torch_tensorrt.Input((1, 3, 224, 224)), # Static NCHW input shape for input #1
        torch_tensorrt.Input(
            min_shape=(1, 224, 224, 3),
            opt_shape=(1, 512, 512, 3),
            max_shape=(1, 1024, 1024, 3),
            dtype=torch.int32
            format=torch.channel_last
        ), # Dynamic input shape for input #2
        torch.randn((1, 3, 224, 244)) # Use an example tensor and let torch_tensorrt infer settings
    ]
    

Keyword Arguments:
  • arg_inputs (Optional[Sequence[Sequence[Any]]]) – Same as inputs. Alias for better understanding with kwarg_inputs.

  • kwarg_inputs (Optional[dict[Any, Any]]) – kwarg inputs to the module forward function.

  • device (Union(Device, torch.device, dict)) –

    Target device for TensorRT engines to run on

    device=torch_tensorrt.Device("dla:1", allow_gpu_fallback=True)
    

  • disable_tf32 (bool) – Force FP32 layers to use traditional as FP32 format vs the default behavior of rounding the inputs to 10-bit mantissas before multiplying, but accumulates the sum using 23-bit mantissas

  • assume_dynamic_shape_support (bool) – Setting this to true enables the converters work for both dynamic and static shapes. Default: False

  • sparse_weights (bool) – Enable sparsity for convolution and fully connected layers.

  • engine_capability (EngineCapability) – Restrict kernel selection to safe gpu kernels or safe dla kernels

  • num_avg_timing_iters (python:int) – Number of averaging timing iterations used to select kernels

  • workspace_size (python:int) – Maximum size of workspace given to TensorRT

  • dla_sram_size (python:int) – Fast software managed RAM used by DLA to communicate within a layer.

  • dla_local_dram_size (python:int) – Host RAM used by DLA to share intermediate tensor data across operations

  • dla_global_dram_size (python:int) – Host RAM used by DLA to store weights and metadata for execution

  • truncate_double (bool) – Truncate weights provided in double (float64) to float32

  • use_complex_decomposition (bool) – Use PyTorch’s upstream complex decomposition (requires torch>=2.14) instead of the legacy hand-rolled complex rewriter. Falls back to the legacy pass when unavailable.

  • require_full_compilation (bool) – Require modules to be compiled end to end or return an error as opposed to returning a hybrid graph where operations that cannot be run in TensorRT are run in PyTorch

  • min_block_size (python:int) – The minimum number of contiguous TensorRT convertible operations in order to run a set of operations in TensorRT

  • torch_executed_ops (Optional[Collection[Target]]) – Set of aten operators that must be run in PyTorch. An error will be thrown if this set is not empty but require_full_compilation is True

  • torch_executed_modules (Optional[Collection[str]]) – Collection of modules that must be run in PyTorch. An error will be thrown if this collection is not empty but require_full_compilation is True

  • pass_through_build_failures (bool) – Error out if there are issues during compilation (only applicable to torch.compile workflows)

  • max_aux_streams (Optional[python:int]) – Maximum streams in the engine

  • version_compatible (bool) – Build the TensorRT engines compatible with future versions of TensorRT (Restrict to lean runtime operators to provide version forward compatibility for the engines)

  • optimization_level – (Optional[int]): Setting a higher optimization level allows TensorRT to spend longer engine building time searching for more optimization options. The resulting engine may have better performance compared to an engine built with a lower optimization level. The default optimization level is 3. Valid values include integers from 0 to the maximum optimization level, which is currently 5. Setting it to be greater than the maximum level results in identical behavior to the maximum level.

  • use_python_runtime – (bool): Deprecated. Kept for backward compatibility; emits a DeprecationWarning when set to True. The Python and C++ runtimes are now merged and the runtime is selected automatically based on whether the C++ Torch-TensorRT runtime is available.

  • use_fast_partitioner – (bool): Use the adjacency based partitioning scheme instead of the global partitioner. Adjacency partitioning is faster but may not be optimal. Use the global paritioner (False) if looking for best performance

  • enable_experimental_decompositions (bool) – Use the full set of operator decompositions. These decompositions may not be tested but serve to make the graph easier to convert to TensorRT, potentially increasing the amount of graphs run in TensorRT.

  • dryrun (bool) – Toggle for “Dryrun” mode, running everything except conversion to TRT and logging outputs

  • hardware_compatible (bool) – Build the TensorRT engines compatible with GPU architectures other than that of the GPU on which the engine was built (currently works for NVIDIA Ampere and newer)

  • target_compute_capabilities (Optional[List[Tuple[python:int, python:int]]]) – Compute capabilities to build for, e.g. [(7, 5)] for Turing. Defaults to None. The compilation then targets the current device, from torch.cuda.get_device_capability(). TensorRT-RTX only. Drives both engine targeting and op partitioning, so ops unsupported on any listed target fall back to PyTorch.

  • timing_cache_path (str) – Path to the timing cache if it exists (or) where it will be saved after compilation. Not used for TensorRT-RTX.

  • lazy_engine_init (bool) – Defer setting up engines until the compilation of all engines is complete. Can allow larger models with multiple graph breaks to compile but can lead to oversubscription of GPU memory at runtime.

  • cache_built_engines (bool) – Whether to save the compiled TRT engines to storage

  • reuse_cached_engines (bool) – Whether to load the compiled TRT engines from storage

  • engine_cache_dir (str) – Directory to store the cached TRT engines

  • engine_cache_size (python:int) – Maximum hard-disk space (bytes) to use for the engine cache, default is 1GB. If the cache exceeds this size, the oldest engines will be removed by default

  • custom_engine_cache (Optional[BaseEngineCache]) – Engine cache instance to use for saving and loading engines. Users can provide their own engine cache by inheriting from BaseEngineCache. If used, engine_cache_dir and engine_cache_size will be ignored.

  • use_fp32_acc (bool) – Enable FP32 accumulation for FP16 matmul layers while retaining FP16 inputs and outputs. When combined with decompose_attention=True, the complete decomposed FP16 scaled dot product attention calculation runs in FP32 and only its final output is cast back to FP16. This option has no effect on FP32 or BF16 inputs.

  • refit_identical_engine_weights (bool) – Refit engines with identical weights. This is useful when the same model is compiled multiple times with different inputs and the weights are the same. This will save time by reusing the same engine for different inputs.

  • strip_engine_weights (bool) – Strip engine weights from the serialized engine. This is useful when the engine is to be deployed in an environment where the weights are not required.

  • immutable_weights (bool) – Build non-refittable engines. This is useful for some layers that are not refittable. If this argument is set to true, strip_engine_weights and refit_identical_engine_weights will be ignored.

  • enable_weight_streaming (bool) – Enable weight streaming.

  • tiling_optimization_level (str) – The optimization level of tiling strategies. A higher level allows TensorRT to spend more time searching for better tiling strategy. We currently support [“none”, “fast”, “moderate”, “full”].

  • l2_limit_for_tiling (python:int) – The target L2 cache usage limit (in bytes) for tiling optimization (default is -1 which means no limit).

  • offload_module_to_cpu (bool) – Offload the module to CPU. This is useful when we need to minimize GPU memory usage.

  • use_distributed_mode_trace (bool) – Using aot_autograd to trace the graph. This is enabled when DTensors or distributed tensors are present in distributed model.

  • decompose_attention (bool) – Whether to decompose attention layers into smaller operations instead of using the attention converters. When combined with use_fp32_acc=True, decomposed FP16 attention keeps its intermediate calculation in FP32 and casts only the final output back to FP16.

  • disabled_constant_fold_exclusions (Collection[str]) – IDs of predefined Torch-TensorRT rules to disable. Rules are registered by lowering implementations and enabled by default; compilation users do not need to register them. Disabling a rule makes the nodes that only it matches foldable. Supported IDs: "attention_mask_arange", "scatter_add_positions". Default is empty.

  • attn_bias_is_causal (Union[bool, str]) – Which attn_bias of efficient SDPA to replace with TensorRT’s built-in causal mask, which is faster and more accurate. True: a bias proven causal, one that depends on no graph input and holds 0 on and below the diagonal and -inf (or the dtype’s lowest value) above, and also a bias that cannot be proven, such as one built from an input, with a dynamic sequence length, or from a module buffer under torch.compile; you assert that it is causal. “proven”: only a bias proven causal, for models whose input-built masks may not be causal, such as masks with padding. A bias that is computed and found not causal is always kept. A mask longer than 4096, or one whose check fails, is not proven: True replaces it and “proven” keeps it. False: never replace. A replaced bias is not read again, so refitting the weights it came from does not change the engine. Default is True.

  • fold_shared_kv_attention (bool) – For attention where all query heads share one key/value head (multi-query attention), compute it against that one head instead of the copies that Hugging Face models make for each query head. TensorRT then runs two plain matrix products instead of its fused attention kernel, which can be faster or slower depending on the GPU, so measure it. In float16 the result can round differently; use_fp32_acc=True reduces that. Only static shapes are rewritten, and scaled_dot_product_attention also needs decompose_attention=True. Both preserved matmuls and the reshape/bmm form produced by AOT tracing under torch.compile are supported for repeat_kv style graphs. In the bmm form, broadcasting a single KV head with enable_gqa=True is not folded. Default is False.

  • deduplicate_weights (bool) – Add a weight that several layers read to the TensorRT network once instead of once per layer, and give the linear layers that share a weight one transposed copy of it. This cuts engine-build host memory for models that reuse weights, such as a denoising loop unrolled into one graph, most of all in float32 builds with disable_tf32=True and in refittable engines. TensorRT may pick other kernels, so results can round differently, and some models need more memory instead, so measure it. Default is False.

  • fallback_pytorch_plugin_ops (bool) – If True, skip converters that insert a TensorRT plugin that runs PyTorch (such as aten.index_put with accumulate=True, which uses Torch-TensorRT’s ScatterAdd plugin, or a generated plugin without an AOT kernel), so the engine runs without PyTorch. This function builds one engine for the whole program, so an operator that only such a converter supports fails to convert. An accumulating aten.index_put with one 1-D index then uses TensorRT’s ScatterElements plugin, which needs no PyTorch. This needs float32, float16, bfloat16, int32 or int64 data. The input and the values it receives (the input’s shape with the indexed dimension set to the index length) must stay under 2**31 elements, so every dynamic size needs an upper bound in the exported program, such as torch.export.Dim(max=…), that keeps them there. Default is False.

  • collapse_high_rank_permutes (bool) – Rewrite a reshape, permute, reshape chain over more than 8 dimensions, TensorRT’s limit, as reshapes and permutes of at most 8 dimensions that give the same values. Window and patch layouts, such as neighborhood attention and the Qwen2-VL image patches, build such chains. This function builds one engine, so without the rewrite such a chain makes the conversion fail. Only static shapes are rewritten. A large engine can need much more time and memory to build and to run, so measure it. Default is False.

  • convert_antialiased_resize (bool) – Convert antialiased bilinear and bicubic resize (F.interpolate with antialias=True) to TensorRT. This function builds one engine, so without this setting such a resize makes the conversion fail. Only static input and output heights and widths and floating-point inputs are converted. Some sizes run slower than in PyTorch, so measure it. Default is False.

  • replace_interleave_stack_with_cat (bool) – Write a stack whose only user is a reshape that merges the new axis into a neighbor axis, as in torch.stack([even, odd], dim=3).flatten(2, 3) for a 2x upsample, as a concatenation of unsqueezed inputs. This can make large, image-sized interleaves faster on GPUs where TensorRT fuses the region around them, but slower where TensorRT runs them as separate layers and adds copies, so measure it. Only programs from torch.export are rewritten, not torch.compile graphs. Default is False.

  • **kwargs – Any,

Returns:

Serialized TensorRT engine, can either be saved to a file or deserialized via TensorRT APIs

Return type:

bytes

Raises:

RuntimeError – if lift_mutable_buffers=True and the model mutates a buffer the engine cannot alias in place, or if a write predicted to be aliased is absent from the built engine’s aliased_io. Either way the buffer would silently never update.

torch_tensorrt.dynamo.cross_compile_for_windows(*args: List[Any], **kwargs: Dict[str, Any]) → Any#
torch_tensorrt.dynamo.save_cross_compiled_exported_program(*args: List[Any], **kwargs: Dict[str, Any]) → Any#
torch_tensorrt.dynamo.load_cross_compiled_exported_program(file_path: str = '') → Any[source]#

Load an ExportedProgram file in Windows which was previously cross compiled in Linux

Parameters:

file_path (str) – Path to file on the disk

Raises:

ValueError – If the api is not called in windows or there is no file or the file is a valid ExportedProgram file

torch_tensorrt.dynamo.refit_module_weights(*args: List[Any], **kwargs: Dict[str, Any]) → Any#

Classes#

class torch_tensorrt.dynamo.low_memory(model_fn: Callable[[], Module], weights: str | PathLike[str], *, offload_module_to_cpu: bool | None = None, enable_resource_partitioning: bool | None = None, cpu_memory_budget: int | None = None, release_weights_after_build: bool | None = None)[source]#

Build a model from its weights file without loading it, with the compile settings that keep its compilation small.

Entering the block builds the model on meta, so it allocates no weights, maps its weights from the safetensors file weights with map_weights, and returns this object, whose model is the model, on the CPU. Every tensor the model reads must be in the file (save_weights_for_mapping writes them all, non-persistent buffers included): a missing parameter or persistent buffer raises, and a buffer left on meta is named in a warning. Export the model with CPU inputs and compile it with torch_tensorrt.dynamo.compile, passing compile_settings; each weight’s pages are read while its engine is built and dropped once its last engine is built. (torch_tensorrt.compile(model, inputs=...) traces with the inputs on the GPU, which a CPU model can’t take.):

with torch_tensorrt.dynamo.low_memory(lambda: MyModel(config), "weights.safetensors") as low_mem:
    exported = torch.export.export(low_mem.model, (example,))
    trt_model = torch_tensorrt.dynamo.compile(
        exported, inputs=[example.cuda()], **low_mem.compile_settings
    )

The keyword arguments are the torch_tensorrt.dynamo.compile settings that lower memory during compilation. compile_settings holds the ones given, with offload_module_to_cpu and enable_resource_partitioning on unless set; a setting left at None otherwise isn’t in it, so compile’s default applies. The settings apply only to the calls they’re passed to.

On leaving, the pages of the model’s mapped parameters are dropped, also on an exception, unless release_weights_after_build=False; buffers, which the model may update, keep theirs. The file stays mapped: the model still reads it, so keep it in place, unchanged, while the model is in use. A write to a mapped weight survives on Linux 5.4 and later; older kernels discard it when its pages are dropped.

Parameters:
  • model_fn – builds the model, without moving it to a device.

  • weights – the safetensors file to map, in the model’s names, dtypes and shapes (save_weights_for_mapping writes one, non-persistent buffers included).

  • offload_module_to_cpu – move the model’s weights to the CPU before the engines are built, so peak GPU memory drops to about the engines’ size while the CPU holds the weights, and move the weights of the parts that run in PyTorch to the GPU afterwards. On by default: the mapped weights are on the CPU already, and a PyTorch part that reads one needs it on the GPU.

  • enable_resource_partitioning – split TensorRT subgraphs into engines small enough to build within cpu_memory_budget. On by default; without a budget, engines are sized to the system’s available memory, so a model that fits isn’t split.

  • cpu_memory_budget – the host memory, in bytes, the compiling process may use, counting what it already uses. Read only with enable_resource_partitioning=True; without a budget, the partitioner sizes engines to the system’s available memory.

  • release_weights_after_build – True also frees weights that aren’t mapped, which the source model then can’t use; False releases nothing, and leaving the block then keeps the pages too. compile’s default drops only mapped weights’ pages, once their last engine is built.

Variables:
  • model – the model, its weights mapped; None until the block is entered.

  • compile_settings – the settings to pass to torch_tensorrt.dynamo.compile.

  • report – what map_weights did (bytes mapped, names missing); None until the block is entered.

Note

low_memory is experimental in 2.15: its API will change in 2.16.

class torch_tensorrt.dynamo.CompilationSettings(workspace_size: int = 0, min_block_size: int = 5, torch_executed_ops: ~typing.Collection[~collections.abc.Callable[[...], ~typing.Any] | str] = <factory>, torch_executed_modules: ~typing.Collection[str] = <factory>, pass_through_build_failures: bool = False, max_aux_streams: int | None = None, version_compatible: bool = False, optimization_level: int | None = None, truncate_double: bool = False, use_complex_decomposition: bool = True, use_fast_partitioner: bool = True, enable_experimental_decompositions: bool = False, device: ~torch_tensorrt._Device.Device = <factory>, require_full_compilation: bool = False, disable_tf32: bool = False, assume_dynamic_shape_support: bool = False, sparse_weights: bool = False, engine_capability: ~torch_tensorrt._enums.EngineCapability = <factory>, num_avg_timing_iters: int = 1, dla_sram_size: int = 1048576, dla_local_dram_size: int = 1073741824, dla_global_dram_size: int = 536870912, dryrun: bool | str = False, hardware_compatible: bool = False, timing_cache_path: str = '/tmp/torch_tensorrt_engine_cache/timing_cache.bin', lazy_engine_init: bool = False, cache_built_engines: bool = False, reuse_cached_engines: bool = False, use_fp32_acc: bool = False, refit_identical_engine_weights: bool = False, strip_engine_weights: bool = False, immutable_weights: bool = True, enable_weight_streaming: bool = False, enable_cross_compile_for_windows: bool = False, tiling_optimization_level: str = 'none', l2_limit_for_tiling: int = -1, target_compute_capabilities: ~typing.List[~typing.Tuple[int, int]] | None = None, use_distributed_mode_trace: bool = False, offload_module_to_cpu: bool = False, enable_autocast: bool = False, autocast_low_precision_type: ~torch_tensorrt._enums.dtype | None = None, autocast_excluded_nodes: ~typing.Collection[str] = <factory>, autocast_excluded_node_names: ~typing.Collection[str] = <factory>, autocast_excluded_ops: ~typing.Collection[~collections.abc.Callable[[...], ~typing.Any] | str] = <factory>, autocast_max_output_threshold: float = 512, autocast_max_depth_of_reduction: int | None = None, autocast_calibration_dataloader: ~torch.utils.data.dataloader.DataLoader | None = None, autocast_model_regions: str = 'keep', _autocast_calibrated_high_precision_nodes: ~typing.FrozenSet[str] | None = None, enable_resource_partitioning: bool = False, cpu_memory_budget: int | None = None, max_partition_size: int | None = None, dynamically_allocate_resources: bool = False, release_weights_after_build: bool | None = None, decompose_attention: bool = False, disabled_constant_fold_exclusions: ~typing.Collection[str] = frozenset({}), attn_bias_is_causal: bool | str = True, fold_shared_kv_attention: bool = False, deduplicate_weights: bool = False, fallback_data_dependent_ops: bool = False, fallback_pytorch_plugin_ops: bool = False, collapse_high_rank_permutes: bool = False, convert_antialiased_resize: bool = False, replace_interleave_stack_with_cat: bool = False, build_route: str | None = None, tune_build_routes: str | None = None, tune_build_route_file: str | None = None, tuning_search: str = 'fast', tuning_timeout_s: int = -1, tuning_cache_file: str | None = None, tuning_continue: bool = False, tuning_dry_run: bool = False, accuracy_threshold: float | None = None, accuracy_algorithm: str = 'l0', accuracy_atol: float = 1e-05, accuracy_rtol: float = 1e-05)[source]#

Compilation settings for Torch-TensorRT Dynamo Paths

Parameters:
  • workspace_size (python:int) – Workspace TRT is allowed to use for the module (0 is default)

  • min_block_size (python:int) – Minimum number of operators per TRT-Engine Block

  • torch_executed_ops (Collection[Target]) – Collection of operations to run in Torch, regardless of converter coverage

  • torch_executed_modules (Collection[str]) – Collection of modules to run in Torch

  • pass_through_build_failures (bool) – Whether to fail on TRT engine build errors (True) or not (False)

  • max_aux_streams (Optional[python:int]) – Maximum number of allowed auxiliary TRT streams for each engine

  • version_compatible (bool) – Provide version forward-compatibility for engine plan files

  • optimization_level (Optional[python:int]) – Builder optimization 0-5, higher levels imply longer build time, searching for more optimization options. TRT defaults to 3

  • truncate_double (bool) – Whether to truncate float64 TRT engine inputs or weights to float32

  • use_fast_partitioner (bool) – Whether to use the fast or global graph partitioning system

  • enable_experimental_decompositions (bool) – Whether to enable all core aten decompositions or only a selected subset of them

  • device (Device) – GPU to compile the model on

  • require_full_compilation (bool) – Whether to require the graph is fully compiled in TensorRT. Only applicable for ir=”dynamo”; has no effect for torch.compile path

  • assume_dynamic_shape_support (bool) – Setting this to true enables the converters work for both dynamic and static shapes. Default: False

  • disable_tf32 (bool) – Whether to disable TF32 computation for TRT layers

  • sparse_weights (bool) – Whether to allow the builder to use sparse weights

  • engine_capability (trt.EngineCapability) – Restrict kernel selection to safe gpu kernels or safe dla kernels

  • num_avg_timing_iters (python:int) – Number of averaging timing iterations used to select kernels

  • dla_sram_size (python:int) – Fast software managed RAM used by DLA to communicate within a layer.

  • dla_local_dram_size (python:int) – Host RAM used by DLA to share intermediate tensor data across operations

  • dla_global_dram_size (python:int) – Host RAM used by DLA to store weights and metadata for execution

  • dryrun (Union[bool, str]) – Toggle “Dryrun” mode, which runs everything through partitioning, short of conversion to TRT Engines. Prints detailed logs of the graph structure and nature of partitioning. Optionally saves the output to a file if a string path is specified

  • hardware_compatible (bool) – Build the TensorRT engines compatible with GPU architectures other than that of the GPU on which the engine was built (currently works for NVIDIA Ampere and newer)

  • target_compute_capabilities (Optional[List[Tuple[python:int, python:int]]]) – Compute capabilities to build for, e.g. [(7, 5)] for Turing. Defaults to None. The compilation then targets the current device, from torch.cuda.get_device_capability(). TensorRT-RTX only – raises on standard TensorRT. Drives both engine targeting and op partitioning: a compiled artifact carries a single partitioning, so an op unsupported on any listed target falls back to PyTorch for all of them.

  • timing_cache_path (str) – Path to the timing cache if it exists (or) where it will be saved after compilation. Not used for TensorRT-RTX (no autotuning).

  • cache_built_engines (bool) – Whether to save the compiled TRT engines to storage

  • reuse_cached_engines (bool) – Whether to load the compiled TRT engines from storage

  • use_fp32_acc (bool) – Enable FP32 accumulation for FP16 matmul layers while retaining FP16 inputs and outputs. When combined with decompose_attention=True, the complete decomposed FP16 scaled dot product attention calculation runs in FP32 and only its final output is cast back to FP16. This option has no effect on FP32 or BF16 inputs.

  • refit_identical_engine_weights (bool) – Whether to refit the engine with identical weights

  • strip_engine_weights (bool) – Whether to strip the engine weights

  • immutable_weights (bool) – Build non-refittable engines. This is useful for some layers that are not refittable. If this argument is set to true, strip_engine_weights and refit_identical_engine_weights will be ignored

  • enable_weight_streaming (bool) – Enable weight streaming.

  • enable_cross_compile_for_windows (bool) – By default this is False means TensorRT engines can only be executed on the same platform where they were built. True will enable cross-platform compatibility which allows the engine to be built on Linux and run on Windows

  • tiling_optimization_level (str) – The optimization level of tiling strategies. A higher level allows TensorRT to spend more time searching for better tiling strategy. We currently support [“none”, “fast”, “moderate”, “full”].

  • l2_limit_for_tiling (python:int) – The target L2 cache usage limit (in bytes) for tiling optimization (default is -1 which means no limit).

  • use_distributed_mode_trace (bool) – Using aot_autograd to trace the graph. This is enabled when DTensors or distributed tensors are present in distributed model

  • enable_autocast (bool) – Whether to enable autocast.

  • autocast_low_precision_type (Optional[Union[torch.dpython:type, dpython:type]]) – The precision to reduce to. We currently support torch.float16 and torch.bfloat16. Default is None, which means no low precision is used.

  • autocast_excluded_nodes (Collection[str]) – The set of regex patterns for the paths of the modules, such as “block.norm”, whose nodes should remain in FP32. Default is [].

  • autocast_excluded_node_names (Collection[str]) – The set of regex patterns for the names of the nodes in the exported graph, such as “layer_norm_1”, that should remain in FP32. A pattern matches from the start of a name, so “layer_norm_1$” matches layer_norm_1 and not layer_norm_10. Default is [].

  • autocast_excluded_ops (Collection[Target]) – The set of targets (ATen ops) that should remain in FP32. Default is [].

  • autocast_max_output_threshold (python:float) – Maximum absolute value for node outputs, nodes with outputs greater than this value will remain in FP32. Needs autocast_calibration_dataloader. Default is 512.

  • autocast_max_depth_of_reduction (Optional[python:int]) – Maximum depth of reduction allowed in low precision. Nodes with higher reduction depths will remain in FP32. This helps prevent excessive accuracy loss in operations particularly sensitive to reduced precision, as higher-depth reductions may amplify computation errors in low precision formats. If not provided, infinity will be used. Needs autocast_calibration_dataloader. Default is None.

  • autocast_calibration_dataloader (Optional[torch.utils.data.DataLoader]) – The dataloader to use for autocast calibration. Default is None.

  • autocast_model_regions (str) – What autocast does with the regions the model itself runs under torch.autocast. “keep” builds them with the precisions that PyTorch autocast gives them. “reclassify” removes their autocast, so that the autocast rules, exclusions and calibration of Torch-TensorRT choose the precision of every op in them too. Regions with enabled=False, which a model uses to keep ops in FP32, keep their autocast. So do regions whose values inlining would change, such as one that writes into a buffer in place, and regions inside a torch.no_grad() block when the model is exported with grad mode on. autocast_excluded_node_names also matches the names that ops have in the regions. Default is “keep”.

  • max_partition_size (Optional[python:int]) – The weight bytes per TensorRT engine the resource partitioner splits to: a finer control than cpu_memory_budget, which it replaces, so the two can’t both be set. The size is then not derived from the budget and the process’s resident memory, so the same model and size always give the same split. Setting it turns resource partitioning on. Default is None.

  • offload_module_to_cpu (bool) – Offload the model to CPU to reduce memory footprint during compilation

  • dynamically_allocate_resources (bool) – Dynamically allocate resources for TensorRT engines

  • release_weights_after_build (Optional[bool]) – Release the memory of each model weight once the last TensorRT engine that reads it is built, and of weights no partition reads before the first build, so compilation holds only the weights still to be built. None (default) releases only weights mapped by map_weights: their pages are dropped and re-read from the file if used again, so the source module keeps working, but an in-place write to a mapped weight is lost. True also frees ordinary weights: the model’s tensors share that memory, so the source module (and an ExportedProgram exported from it) must not be used after compilation, even if compilation fails. False releases nothing. Weights read by subgraphs that run in PyTorch are kept. Applies to torch_tensorrt.dynamo.compile; no effect under torch.compile (parameters arrive as graph inputs). MutableTorchTensorRTModule uses False.

  • decompose_attention (bool) – Whether to decompose attention layers into smaller operations instead of using the attention converters. When combined with use_fp32_acc=True, decomposed FP16 attention keeps its intermediate calculation in FP32 and casts only the final output back to FP16.

  • disabled_constant_fold_exclusions (Collection[str]) – IDs of predefined Torch-TensorRT rules to disable. Rules are registered by lowering implementations and enabled by default; compilation users do not need to register them. Disabling a rule makes the nodes that only it matches foldable. Supported IDs: "attention_mask_arange", "scatter_add_positions". Default is empty.

  • attn_bias_is_causal (Union[bool, str]) – Which attn_bias of efficient SDPA to replace with TensorRT’s built-in causal mask, which is faster and more accurate. True: a bias proven causal, one that depends on no graph input and holds 0 on and below the diagonal and -inf (or the dtype’s lowest value) above, and also a bias that cannot be proven, such as one built from an input, with a dynamic sequence length, or from a module buffer under torch.compile; you assert that it is causal. “proven”: only a bias proven causal, for models whose input-built masks may not be causal, such as masks with padding. A bias that is computed and found not causal is always kept. A mask longer than 4096, or one whose check fails, is not proven: True replaces it and “proven” keeps it. False: never replace. A replaced bias is not read again, so refitting the weights it came from does not change the engine. Default is True.

  • fold_shared_kv_attention (bool) – For attention where all query heads share one key/value head (multi-query attention), compute it against that one head instead of the copies that Hugging Face models make for each query head. TensorRT then runs two plain matrix products instead of its fused attention kernel, which can be faster or slower depending on the GPU, so measure it. In float16 the result can round differently; use_fp32_acc=True reduces that. Only static shapes are rewritten, and scaled_dot_product_attention also needs decompose_attention=True. Both preserved matmuls and the reshape/bmm form produced by AOT tracing under torch.compile are supported for repeat_kv style graphs. In the bmm form, broadcasting a single KV head with enable_gqa=True is not folded. Default is False.

  • deduplicate_weights (bool) – Add a weight that several layers read to the TensorRT network once instead of once per layer, and give the linear layers that share a weight one transposed copy of it. This cuts engine-build host memory for models that reuse weights, such as a denoising loop unrolled into one graph, most of all in float32 builds with disable_tf32=True and in refittable engines. TensorRT may pick other kernels, so results can round differently, and some models need more memory instead, so measure it. Under torch.compile the weights are engine inputs, not constants, so they are not shared. Default is False.

  • fallback_data_dependent_ops (bool) – If True, operators whose converters require a TensorRT output allocator (i.e. data-dependent output shapes, such as nonzero) are added to torch_executed_ops and run in PyTorch instead of being lowered into a TensorRT engine. This is useful when targeting runtimes that cannot consume a TensorRT output allocator. Default is False.

  • fallback_pytorch_plugin_ops (bool) – If True, operators whose converters insert a TensorRT plugin that runs PyTorch (such as aten.index_put with accumulate=True, which uses Torch-TensorRT’s ScatterAdd plugin, or a generated plugin without an AOT kernel) run in PyTorch instead of being lowered into a TensorRT engine. An accumulating aten.index_put with one 1-D index then stays in TensorRT through TensorRT’s ScatterElements plugin, which needs no PyTorch. This needs float32, float16, bfloat16, int32 or int64 data. The input and the values it receives (the input’s shape with the indexed dimension set to the index length) must stay under 2**31 elements, so every dynamic size needs an upper bound in the exported program, such as torch.export.Dim(max=…), that keeps them there. Otherwise the operator runs in PyTorch. Use it for runtimes without PyTorch, such as the ExecuTorch TensorRT runtime. Default is False.

  • collapse_high_rank_permutes (bool) – Rewrite a reshape, permute, reshape chain over more than 8 dimensions, TensorRT’s limit, as reshapes and permutes of at most 8 dimensions that give the same values. Window and patch layouts, such as neighborhood attention and the Qwen2-VL image patches, build such chains. Left in PyTorch, each one splits the graph, and the partitioner can then hand many large tensors across one engine boundary. Only static shapes are rewritten. Fewer, larger engines can need much more time and memory to build and to run, so measure it. Default is False.

  • convert_antialiased_resize (bool) – Run antialiased bilinear and bicubic resize (F.interpolate with antialias=True) in TensorRT instead of PyTorch, so it does not split the graph. Only static input and output heights and widths and floating-point inputs are converted. Some sizes run slower than in PyTorch, so measure it. Default is False.

  • replace_interleave_stack_with_cat (bool) – Write a stack whose only user is a reshape that merges the new axis into a neighbor axis, as in torch.stack([even, odd], dim=3).flatten(2, 3) for a 2x upsample, as a concatenation of unsqueezed inputs. This can make large, image-sized interleaves faster on GPUs where TensorRT fuses the region around them, but slower where TensorRT runs them as separate layers and adds copies, so measure it. Only programs from torch.export are rewritten, not torch.compile graphs. Default is False.

  • build_route (Optional[str]) – TensorRT Global Performance Tuning build route string (space-separated “-knob=value” tokens). None uses the default route. This feature requires TensorRT with Global Performance Tuning enabled; currently unavailable on TensorRT-RTX / Windows.

  • tune_build_routes (Optional[str]) – Build-route expression for an autotuning sweep that runs each candidate in a spawned child process (e.g. “-slice_fusion=[on|off] -kgen:codegen:cuda_tile=[0|1|2|3]”). None disables tuning.

  • tune_build_route_file (Optional[str]) – Path to a file with one route token per line (same as “tune_build_routes”). Mutually exclusive with “tune_build_routes”.

  • tuning_search (str) – Search algorithm: “fast”, “full”, or “mixed”. Default is “fast”. “fast” runs a baseline with all knobs at default, then varies one knob at a time. “full” runs a full grid search. “mixed” runs “fast” first, and then “full” only on knobs that improved performance.

  • tuning_timeout_s (python:int) – Time budget for the entire tuning process. The current iteration finishes before the loop stops. Use -1 (default) to disable the timeout. Helpful for capping large “full” sweeps.

  • tuning_cache_file (Optional[str]) – JSONL path for tuning results / resume.

  • tuning_continue (bool) – Resume an interrupted sweep from “tuning_cache_file”. When it is True, “tuning_cache_file” must be provided; “tune_build_routes”, “tune_build_route_file”, and “tuning_dry_run” must not be provided.

  • tuning_dry_run (bool) – Enumerate candidate routes without building engines.

  • accuracy_threshold (Optional[python:float]) – Max allowed accuracy loss between the engine outputs and eager Torch reference outputs; routes with accuracy loss above this threshold are excluded from best-engine selection. “None” skips accuracy checks.

  • accuracy_algorithm (str) – Loss metric: “l0”, “l1”, “l2”, “lInf”, or “cos” (lower is better).

  • accuracy_atol (python:float) – Absolute tolerance for “accuracy_algorithm=”l0””.

  • accuracy_rtol (python:float) – Relative tolerance for “accuracy_algorithm=”l0””.

Tools#

Weight mapping: compile from a safetensors checkpoint’s weights in place (see the resource management guide).

Note

These tools are experimental in 2.15: their API will change in 2.16.