Multiple Optimization Profiles: Prefill vs Decode for Gemma-3#
Autoregressive LLMs run in two very different shape regimes that share one set of weights (and ideally one engine):
prefill: the prompt is processed in one shot, so the sequence length
seqis large, anddecode: tokens are generated one at a time, so
seq == 1.
A single dynamic range seq in [1, max] works, but TensorRT can only tune kernels for one opt point. Tuning for the prefill length leaves decode (the latency-critical, most-frequently-executed phase) running on kernels picked for a sequence it never sees.
torch_tensorrt.Input(profiles=[...]) declares N optimization profiles on a single input. The engine is built once (a single torch.export over the union of all profiles), each profile gets its own TensorRT kernel tuning, and you select the active profile per call (by index, or "auto").
This example compiles google/gemma-3-1b-it once into a two-profile engine and then runs the same engine two ways: every call on the prefill profile (which accepts seq == 1 as well) versus each phase on its own profile. One engine, one set of weights; the only difference is which profile is active when the call runs. What that isolates is what the active profile is worth, holding the engine fixed. It is not a stand-in for a separately
compiled single-profile engine: that is a different build, and on the same A40 it makes decode look 1.14x faster rather than the 1.30x measured here. The two-engine pairing is in the tutorial <multi_optimization_profiles_tutorial>.
Note
google/gemma-3-1b-it is a gated model: you must accept its license on the Hugging Face Hub and authenticate (hf auth login or the HF_TOKEN environment variable) before running this example. It downloads ~2 GB of weights on first use and requires a CUDA GPU.
Note
This uses the Ahead-Of-Time (AOT) torch.export + dynamo.compile path. Runtime profile selection works with whichever TensorRT runtime (C++ or Python) the installed Torch-TensorRT build provides.
Imports and Setup#
export_llm, a reusable helper that traces a decoder over a dynamic sequence length, lives under tools/llm in the Torch-TensorRT repo, so we add that directory to sys.path.
[ ]:
import sys
import timeit
from pathlib import Path
import torch
import torch_tensorrt
REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT / "tools" / "llm"))
MODEL_ID = "google/gemma-3-1b-it"
DEVICE = torch.device("cuda:0")
# The two regimes we benchmark.
MAX_SEQ = 256 # largest prompt the engine must support
PREFILL_SEQ = 128
DECODE_SEQ = 1
DECODE_IDX, PREFILL_IDX = 0, 1
Load the Model#
Load with use_cache=False (this example recomputes over the full sequence rather than using a KV cache, which keeps the export simple). The sdpa attention implementation makes HuggingFace emit scaled_dot_product_attention, which Torch-TensorRT converts to a single TensorRT attention layer.
[ ]:
def load_model():
from transformers import AutoModelForCausalLM
with torch.no_grad():
model = (
AutoModelForCausalLM.from_pretrained(
MODEL_ID,
use_cache=False,
attn_implementation="sdpa",
ignore_mismatched_sizes=True,
)
.eval()
.cuda()
.to(torch.float16)
)
return model
try:
model = load_model()
except Exception as e: # gated/no-auth/no-GPU environments (e.g. CI docs build)
print(f"Skipping example: could not load {MODEL_ID} ({type(e).__name__}: {e}).")
print("Accept the license and authenticate (hf auth login / HF_TOKEN) to run.")
sys.exit(0)
def make_inputs(seq_len: int):
ids = torch.randint(1, 10000, (1, seq_len), dtype=torch.int64, device=DEVICE)
position_ids = torch.arange(seq_len, device=DEVICE).unsqueeze(0)
return ids, position_ids
Declaring the Optimization Profiles#
profiles is an ordered list; the list index is the optimization-profile index used at runtime. Both model inputs (input_ids and position_ids) are dynamic over seq, so each gets a profiled Input with identical profiles:
index
0-> decode:seqpinned to 1 (a fully static profile)index
1-> prefill:seqin[1, MAX_SEQ], tuned atPREFILL_SEQ
Profile order matters for auto-selection, but only on a rescan. Auto-selection is sticky: it keeps the active profile whenever that profile still accepts the input (switching invalidates the captured CUDA graph and forces shape re-inference), and only when it does not does it rescan and take the lowest index that fits. So declaring decode first wins the seq == 1 overlap on a rescan, but it does not claw seq == 1 back from prefill: prefill’s range is [1, MAX_SEQ], which
still fits, so an alternating prefill/decode loop stays on prefill. Pin the profile (below) to make decode run under its own.
[ ]:
profiles = [
{"min_shape": (1, 1), "opt_shape": (1, 1), "max_shape": (1, 1)}, # decode
{
"min_shape": (1, 1),
"opt_shape": (1, PREFILL_SEQ),
"max_shape": (1, MAX_SEQ),
}, # prefill
]
multi_profile_inputs = [
torch_tensorrt.Input(dtype=torch.int64, profiles=profiles), # input_ids
torch_tensorrt.Input(dtype=torch.int64, profiles=profiles), # position_ids
]
Export Once, Compile Once#
export_llm traces the model over a dynamic seq range, and one compile turns that into one engine holding both profiles. There is no second compile: the prefill profile already accepts seq == 1, so pinning every call to it is the comparison point, and both sides of the measurement below then share one engine and one set of weights.
[ ]:
from utils import export_llm # noqa: E402
example_ids, _ = make_inputs(PREFILL_SEQ)
with torch.inference_mode():
exported = export_llm(model, example_ids, min_seq_len=1, max_seq_len=MAX_SEQ)
print("Compiling multi-profile engine (decode + prefill) ...")
# ``offload_module_to_cpu`` must stay False here: it is currently incompatible
# with the multi-profile ``Input(profiles=...)`` path (CPU/CUDA device mismatch).
trt_model = torch_tensorrt.dynamo.compile(
exported,
arg_inputs=multi_profile_inputs,
use_fp32_acc=True,
disable_tf32=True,
offload_module_to_cpu=False,
min_block_size=1,
require_full_compilation=True,
device=DEVICE,
)
Correctness#
FP16 logits over Gemma’s 262K-token vocabulary are noisy, so we compare the predicted token (argmax) rather than raw logits.
[ ]:
def logits(out):
return (out.logits if hasattr(out, "logits") else out).float()
from torch_tensorrt.runtime import optimization_profile # noqa: E402
decode_ids, decode_pos = make_inputs(DECODE_SEQ)
prefill_ids, prefill_pos = make_inputs(PREFILL_SEQ)
with torch.inference_mode():
ref_decode = logits(model(decode_ids, position_ids=decode_pos))
ref_prefill = logits(model(prefill_ids, position_ids=prefill_pos))
with optimization_profile(trt_model, DECODE_IDX):
trt_decode = logits(trt_model(decode_ids, decode_pos))
with optimization_profile(trt_model, PREFILL_IDX):
trt_prefill = logits(trt_model(prefill_ids, prefill_pos))
def top1_match(a, b):
return (a.argmax(-1) == b.argmax(-1)).float().mean().item()
print(f"decode top-1 token match vs eager: {top1_match(trt_decode, ref_decode):.1%}")
print(f"prefill top-1 token match vs eager: {top1_match(trt_prefill, ref_prefill):.1%}")
Latency Comparison#
Decode is timed twice against the one engine: once on the prefill profile and once on its own. The profile is pinned around the whole loop rather than per call, which is the realistic serving pattern and keeps profile switches out of the measurement. We report the min over several rounds to reduce noise.
[ ]:
def benchmark(run, iters: int = 50, warmup: int = 20, rounds: int = 3) -> float:
for _ in range(warmup):
run()
torch.cuda.synchronize()
best = float("inf")
for _ in range(rounds):
start = timeit.default_timer()
for _ in range(iters):
run()
torch.cuda.synchronize()
best = min(best, (timeit.default_timer() - start) / iters * 1000) # ms/call
return best
with torch.inference_mode():
with optimization_profile(trt_model, PREFILL_IDX):
decode_on_prefill = benchmark(lambda: trt_model(decode_ids, decode_pos))
prefill_on_prefill = benchmark(lambda: trt_model(prefill_ids, prefill_pos))
with optimization_profile(trt_model, DECODE_IDX):
decode_on_decode = benchmark(lambda: trt_model(decode_ids, decode_pos))
Results. Decode is the win: the decode profile pins seq to 1, so TensorRT specializes that path instead of serving it from kernels tuned for the long prefill length. Prefill appears once because the decode profile does not accept a 128-token input at all – prefill has only one profile it can run on, so it is the same call in both scenarios.
[ ]:
print("\nPer-call latency (ms), batch=1")
print(f"{'call':<24}{'active profile':>18}{'ms':>10}")
print("-" * 52)
print(f"{f'decode (seq={DECODE_SEQ})':<24}{'prefill':>18}{decode_on_prefill:>10.3f}")
print(f"{f'decode (seq={DECODE_SEQ})':<24}{'decode':>18}{decode_on_decode:>10.3f}")
print(f"{f'prefill (seq={PREFILL_SEQ})':<24}{'prefill':>18}{prefill_on_prefill:>10.3f}")
print(
f"\nGiving decode its own profile: {decode_on_prefill / decode_on_decode:.2f}x "
f"faster per token ({decode_on_prefill - decode_on_decode:+.3f} ms)"
)
Summary#
Declare
Nprofiles on anInputwithprofiles=[{min_shape, opt_shape, max_shape}, ...](one per dynamic model input – hereinput_idsandposition_ids).One export + one engine; each profile gets its own TensorRT kernel tuning.
A profile whose range covers the other regime is the comparison point: pinning every call to the prefill profile isolates what the active profile is worth on one engine. A separately compiled single-profile engine is a different build and lands elsewhere – see the tutorial for that pairing.
Select at runtime by index (
optimization_profile(m, i)) or with"auto", which keeps the active profile while it still fits and rescans from index 0 only once it does not.Dedicating a static
seq == 1profile to decode lets TensorRT tune that latency-critical path independently of the prefill length.
[ ]:
print("Done.")