Low CPU Memory Compilation Example#
This example demonstrates compiling a model with a bounded CPU (host) memory budget using Torch-TensorRT Dynamo. Limiting host RAM use is helpful on memory-constrained machines or when compiling very large models.
Key notes: - The toy model below has roughly 430 MB of parameters. We set the CPU memory budget to 2 GiB. At compile time, only about 900 MB of host RAM may remain available. Building an engine can take up to four times its weights, so each engine may hold about 900 / 4 = 225 MB. The model is partitioned into three subgraphs: one for each convolution (about 150 MB each) and one for the linear layer (about 130 MB).
The per-engine size depends on the host memory in use when partitioning starts, so the split can change from one run to the next. For finer control, set the size directly with
max_partition_size, in bytes of weights per engine, instead ofcpu_memory_budget. With200 * 1024**2this model splits into the same three engines on every run.320 * 1024**2puts both convolutions in one engine and the linear layer in another, and a size above the model’s 430 MB keeps one engine.Performance impact varies by model. When the number of TensorRT engines created is small, the impact is typically minimal.
[ ]:
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch_tensorrt as torchtrt
from torch_tensorrt.dynamo.conversion import CompilationSettings
class net(nn.Module):
def __init__(self):
super().__init__()
# Intentionally large layers to stress host memory during compilation.
self.conv1 = nn.Conv2d(1024, 4096, 3, padding=1)
self.bn1 = nn.BatchNorm2d(4096)
self.conv2 = nn.Conv2d(4096, 1024, 3, padding=1)
self.bn2 = nn.BatchNorm2d(1024)
self.fc1 = nn.Linear(1024 * 56 * 56, 10)
def forward(self, x):
x = self.conv1(x)
x = self.bn1(x)
x = F.relu(x)
x = F.max_pool2d(x, (2, 2))
x = self.conv2(x)
x = self.bn2(x)
x = F.relu(x)
x = F.max_pool2d(x, (2, 2))
x = torch.flatten(x, 1)
return self.fc1(x)
model = net().eval()
model.to("cuda")
inputs = [torch.randn((1, 1024, 224, 224)).to("cuda")]
compilation_options = {
"min_block_size": 1,
"immutable_weights": True,
"reuse_cached_engines": False,
"enable_resource_partitioning": True,
"cpu_memory_budget": 2 * 1024 * 1024 * 1024, # 2 GiB in bytes
}
settings = CompilationSettings(**compilation_options)
with torchtrt.dynamo.Debugger(
log_level="debug",
logging_dir="/home/profile/logging/moe",
engine_builder_monitor=False,
):
exp_program = torch.export.export(model, tuple(inputs))
trt_gm = torchtrt.dynamo.compile(
exp_program,
inputs=inputs,
**compilation_options,
)
# Expect three back-to-back TensorRT engines due to partitioning under the memory budget.
print(trt_gm)
"""
You should be able to see three back-to-back TensorRT engines in the graph
Graph Structure:
Inputs: List[Tensor: (1, 1024, 224, 224)@float32]
...
TRT Engine #1 - Submodule name: _run_on_acc_0_resource_split_0
Engine Inputs: List[Tensor: (1, 1024, 224, 224)@float32]
Number of Operators in Engine: 5
Engine Outputs: List[Tensor: (1, 4096, 112, 112)@float32]
...
TRT Engine #2 - Submodule name: _run_on_acc_0_resource_split_1
Engine Inputs: List[Tensor: (1, 4096, 112, 112)@float32]
Number of Operators in Engine: 4
Engine Outputs: List[Tensor: (1, 1024, 112, 112)@float32]
...
TRT Engine #3 - Submodule name: _run_on_acc_0_resource_split_2
Engine Inputs: List[Tensor: (1, 1024, 112, 112)@float32]
Number of Operators in Engine: 3
Engine Outputs: List[Tensor: (1, 10)@float32]
...
Outputs: List[Tensor: (1, 10)@float32]
------------------------- Aggregate Stats -------------------------
Average Number of Operators per TRT Engine: 4.0
Most Operators in a TRT Engine: 5
********** Recommendations **********
- For minimal graph segmentation, select min_block_size=5 which would generate 1 TRT engine(s)
- For moderate graph segmentation, select min_block_size=4 which would generate 2 TRT engine(s)
- The current level of graph segmentation is equivalent to selecting min_block_size=3 which generates 3 TRT engine(s)
GraphModule(
(_run_on_acc_0_resource_split_0): TorchTensorRTModule()
(_run_on_acc_0_resource_split_1): TorchTensorRTModule()
(_run_on_acc_0_resource_split_2): TorchTensorRTModule()
)
def forward(self, x):
x, = fx_pytree.tree_flatten_spec(([x], {}), self._in_spec)
_run_on_acc_0_resource_split_0 = self._run_on_acc_0_resource_split_0(x); x = None
_run_on_acc_0_resource_split_1 = self._run_on_acc_0_resource_split_1(_run_on_acc_0_resource_split_0); _run_on_acc_0_resource_split_0 = None
_run_on_acc_0_resource_split_2 = self._run_on_acc_0_resource_split_2(_run_on_acc_0_resource_split_1); _run_on_acc_0_resource_split_1 = None
return pytree.tree_unflatten((_run_on_acc_0_resource_split_2,), self._out_spec)
)
"""