Multiple Optimization Profiles: Prefill vs Decode for Gemma-3#

Autoregressive LLMs run in two very different shape regimes that share one set of weights (and ideally one engine):

  • prefill: the prompt is processed in one shot, so the sequence length seq is large, and

  • decode: tokens are generated one at a time, so seq == 1.

A single dynamic range seq in [1, max] works, but TensorRT can only tune kernels for one opt point. Tuning for the prefill length leaves decode (the latency-critical, most-frequently-executed phase) running on kernels picked for a sequence it never sees.

torch_tensorrt.Input(profiles=[...]) declares N optimization profiles on a single input. The engine is built once (a single torch.export over the union of all profiles), each profile gets its own TensorRT kernel tuning, and you select the active profile per call (by index, or "auto").

This example compiles google/gemma-3-1b-it once into a two-profile engine and then runs the same engine two ways: every call on the prefill profile (which accepts seq == 1 as well) versus each phase on its own profile. One engine, one set of weights; the only difference is which profile is active when the call runs. What that isolates is what the active profile is worth, holding the engine fixed. It is not a stand-in for a separately compiled single-profile engine: that is a different build, and on the same A40 it makes decode look 1.14x faster rather than the 1.30x measured here. The two-engine pairing is in the tutorial <multi_optimization_profiles_tutorial>.

Note

google/gemma-3-1b-it is a gated model: you must accept its license on the Hugging Face Hub and authenticate (hf auth login or the HF_TOKEN environment variable) before running this example. It downloads ~2 GB of weights on first use and requires a CUDA GPU.

Note

This uses the Ahead-Of-Time (AOT) torch.export + dynamo.compile path. Runtime profile selection works with whichever TensorRT runtime (C++ or Python) the installed Torch-TensorRT build provides.

Imports and Setup#

export_llm, a reusable helper that traces a decoder over a dynamic sequence length, lives under tools/llm in the Torch-TensorRT repo, so we add that directory to sys.path.

[ ]:
import sys
import timeit
from pathlib import Path

import torch
import torch_tensorrt

REPO_ROOT = Path(__file__).resolve().parents[2]
sys.path.insert(0, str(REPO_ROOT / "tools" / "llm"))

MODEL_ID = "google/gemma-3-1b-it"
DEVICE = torch.device("cuda:0")

# The two regimes we benchmark.
MAX_SEQ = 256  # largest prompt the engine must support
PREFILL_SEQ = 128
DECODE_SEQ = 1
DECODE_IDX, PREFILL_IDX = 0, 1

Load the Model#

Load with use_cache=False (this example recomputes over the full sequence rather than using a KV cache, which keeps the export simple). The sdpa attention implementation makes HuggingFace emit scaled_dot_product_attention, which Torch-TensorRT converts to a single TensorRT attention layer.

[ ]:
def load_model():
    from transformers import AutoModelForCausalLM

    with torch.no_grad():
        model = (
            AutoModelForCausalLM.from_pretrained(
                MODEL_ID,
                use_cache=False,
                attn_implementation="sdpa",
                ignore_mismatched_sizes=True,
            )
            .eval()
            .cuda()
            .to(torch.float16)
        )
    return model


try:
    model = load_model()
except Exception as e:  # gated/no-auth/no-GPU environments (e.g. CI docs build)
    print(f"Skipping example: could not load {MODEL_ID} ({type(e).__name__}: {e}).")
    print("Accept the license and authenticate (hf auth login / HF_TOKEN) to run.")
    sys.exit(0)


def make_inputs(seq_len: int):
    ids = torch.randint(1, 10000, (1, seq_len), dtype=torch.int64, device=DEVICE)
    position_ids = torch.arange(seq_len, device=DEVICE).unsqueeze(0)
    return ids, position_ids

Declaring the Optimization Profiles#

profiles is an ordered list; the list index is the optimization-profile index used at runtime. Both model inputs (input_ids and position_ids) are dynamic over seq, so each gets a profiled Input with identical profiles:

  • index 0 -> decode: seq pinned to 1 (a fully static profile)

  • index 1 -> prefill: seq in [1, MAX_SEQ], tuned at PREFILL_SEQ

Profile order matters for auto-selection, but only on a rescan. Auto-selection is sticky: it keeps the active profile whenever that profile still accepts the input (switching invalidates the captured CUDA graph and forces shape re-inference), and only when it does not does it rescan and take the lowest index that fits. So declaring decode first wins the seq == 1 overlap on a rescan, but it does not claw seq == 1 back from prefill: prefill’s range is [1, MAX_SEQ], which still fits, so an alternating prefill/decode loop stays on prefill. Pin the profile (below) to make decode run under its own.

[ ]:
profiles = [
    {"min_shape": (1, 1), "opt_shape": (1, 1), "max_shape": (1, 1)},  # decode
    {
        "min_shape": (1, 1),
        "opt_shape": (1, PREFILL_SEQ),
        "max_shape": (1, MAX_SEQ),
    },  # prefill
]
multi_profile_inputs = [
    torch_tensorrt.Input(dtype=torch.int64, profiles=profiles),  # input_ids
    torch_tensorrt.Input(dtype=torch.int64, profiles=profiles),  # position_ids
]

Export Once, Compile Once#

export_llm traces the model over a dynamic seq range, and one compile turns that into one engine holding both profiles. There is no second compile: the prefill profile already accepts seq == 1, so pinning every call to it is the comparison point, and both sides of the measurement below then share one engine and one set of weights.

[ ]:
from utils import export_llm  # noqa: E402

example_ids, _ = make_inputs(PREFILL_SEQ)
with torch.inference_mode():
    exported = export_llm(model, example_ids, min_seq_len=1, max_seq_len=MAX_SEQ)

print("Compiling multi-profile engine (decode + prefill) ...")
# ``offload_module_to_cpu`` must stay False here: it is currently incompatible
# with the multi-profile ``Input(profiles=...)`` path (CPU/CUDA device mismatch).
trt_model = torch_tensorrt.dynamo.compile(
    exported,
    arg_inputs=multi_profile_inputs,
    use_fp32_acc=True,
    disable_tf32=True,
    offload_module_to_cpu=False,
    min_block_size=1,
    require_full_compilation=True,
    device=DEVICE,
)

Correctness#

FP16 logits over Gemma’s 262K-token vocabulary are noisy, so we compare the predicted token (argmax) rather than raw logits.

[ ]:
def logits(out):
    return (out.logits if hasattr(out, "logits") else out).float()


from torch_tensorrt.runtime import optimization_profile  # noqa: E402

decode_ids, decode_pos = make_inputs(DECODE_SEQ)
prefill_ids, prefill_pos = make_inputs(PREFILL_SEQ)

with torch.inference_mode():
    ref_decode = logits(model(decode_ids, position_ids=decode_pos))
    ref_prefill = logits(model(prefill_ids, position_ids=prefill_pos))

    with optimization_profile(trt_model, DECODE_IDX):
        trt_decode = logits(trt_model(decode_ids, decode_pos))
    with optimization_profile(trt_model, PREFILL_IDX):
        trt_prefill = logits(trt_model(prefill_ids, prefill_pos))


def top1_match(a, b):
    return (a.argmax(-1) == b.argmax(-1)).float().mean().item()


print(f"decode  top-1 token match vs eager: {top1_match(trt_decode, ref_decode):.1%}")
print(f"prefill top-1 token match vs eager: {top1_match(trt_prefill, ref_prefill):.1%}")

Latency Comparison#

Decode is timed twice against the one engine: once on the prefill profile and once on its own. The profile is pinned around the whole loop rather than per call, which is the realistic serving pattern and keeps profile switches out of the measurement. We report the min over several rounds to reduce noise.

[ ]:
def benchmark(run, iters: int = 50, warmup: int = 20, rounds: int = 3) -> float:
    for _ in range(warmup):
        run()
    torch.cuda.synchronize()
    best = float("inf")
    for _ in range(rounds):
        start = timeit.default_timer()
        for _ in range(iters):
            run()
        torch.cuda.synchronize()
        best = min(best, (timeit.default_timer() - start) / iters * 1000)  # ms/call
    return best


with torch.inference_mode():
    with optimization_profile(trt_model, PREFILL_IDX):
        decode_on_prefill = benchmark(lambda: trt_model(decode_ids, decode_pos))
        prefill_on_prefill = benchmark(lambda: trt_model(prefill_ids, prefill_pos))
    with optimization_profile(trt_model, DECODE_IDX):
        decode_on_decode = benchmark(lambda: trt_model(decode_ids, decode_pos))

Results. Decode is the win: the decode profile pins seq to 1, so TensorRT specializes that path instead of serving it from kernels tuned for the long prefill length. Prefill appears once because the decode profile does not accept a 128-token input at all – prefill has only one profile it can run on, so it is the same call in both scenarios.

[ ]:
print("\nPer-call latency (ms), batch=1")
print(f"{'call':<24}{'active profile':>18}{'ms':>10}")
print("-" * 52)
print(f"{f'decode (seq={DECODE_SEQ})':<24}{'prefill':>18}{decode_on_prefill:>10.3f}")
print(f"{f'decode (seq={DECODE_SEQ})':<24}{'decode':>18}{decode_on_decode:>10.3f}")
print(f"{f'prefill (seq={PREFILL_SEQ})':<24}{'prefill':>18}{prefill_on_prefill:>10.3f}")
print(
    f"\nGiving decode its own profile: {decode_on_prefill / decode_on_decode:.2f}x "
    f"faster per token ({decode_on_prefill - decode_on_decode:+.3f} ms)"
)

Summary#

  • Declare N profiles on an Input with profiles=[{min_shape, opt_shape, max_shape}, ...] (one per dynamic model input – here input_ids and position_ids).

  • One export + one engine; each profile gets its own TensorRT kernel tuning.

  • A profile whose range covers the other regime is the comparison point: pinning every call to the prefill profile isolates what the active profile is worth on one engine. A separately compiled single-profile engine is a different build and lands elsewhere – see the tutorial for that pairing.

  • Select at runtime by index (optimization_profile(m, i)) or with "auto", which keeps the active profile while it still fits and rescans from index 0 only once it does not.

  • Dedicating a static seq == 1 profile to decode lets TensorRT tune that latency-critical path independently of the prefill length.

[ ]:
print("Done.")