What Actually Happens When TensorRT-LLM Runs a Model?
A call to LLM.generate() or trtllm-serve tokenizes the prompt and submits it to a per-rank PyExecutor worker running an asynchronous loop with the Overlap Scheduler. Each step, C++-bound CapacityScheduler and MicroBatchScheduler pick active context and generation requests while the C++-bound KVCacheManager allocates paged KV-cache blocks. ModelEngine delegates the batch to a native PyTorch model (PyTorch is now TensorRT-LLM's sole execution backend; `trtllm-build` engines were removed). With Piecewise CUDA Graph enabled, TorchDynamo traces the model into Torch IR and TensorRT-LLM's custom torch.compile backend lowers it via aot_module_simplified to ATen IR—fusing AllReduce + RMSNorm and scheduling multiple CUDA streams without using TorchInductor—then replays Piecewise CUDA Graphs around in-place TrtllmAttention or FlashInferAttention kernels.
Showing torch.compile with Piecewise CUDA Graphs (prefill_cuda_graph_backend: piecewise) with the default TrtllmAttention backend: 13 stages, 13 steps.
Your program
Runtime
Framework core
Kernel
Compiler
Hardware
The selected path, step by step
LLM API / serve tokenizes and submits request to PyExecutor worker: LLM.generate() or trtllm-serve tokenizes the prompt and queues the request for the rank's PyExecutor worker.
PyExecutor worker schedules active requests via C++ batch scheduler: At each iteration of its background loop, PyExecutor invokes SimpleScheduler (combining CapacityScheduler and MicroBatchScheduler).
C++ batch scheduler checks capacity and reserves blocks in C++ KVCacheManager: CapacityScheduler queries KVCacheManager for available blocks and PyExecutor calls prepare_resources to allocate paged KV-cache blocks for the batch.
C++ KVCacheManager passes scheduled batch & KV buffers to ModelEngine & runner: PyExecutor invokes ModelEngine.forward with the scheduled context and generation requests and their allocated KV-cache resources.
ModelEngine & runner runs forward pass on PyTorch model: ScheduledModelRunner prepares the input tensors and executes the PyTorch model's forward pass.
PyTorch model invokes attention through Attention custom op: Each transformer layer prepares AttentionMetadata and calls the wrapped attention custom op.
PyTorch model traced in fullgraph mode by TorchDynamo (FX): Enabling prefill_cuda_graph_backend: piecewise enables torch.compile, which traces the decoder model with TorchDynamo into a Torch IR FX graph.
TorchDynamo (FX) lowered via aot_module_simplified to TRT-LLM ATen IR: TensorRT-LLM's custom backend lowers the FX graph to ATen IR, fuses AllReduce + Residual + RMSNorm, removes mutation-wrapper copies, and inserts multi-stream scheduling ops.
TRT-LLM ATen IR partitioned at attention into Piecewise CUDA Graph: Non-capturable attention is split out while the surrounding fused ATen/custom-op segments are captured per token-count bucket as Piecewise CUDA Graphs.
Attention custom op dispatches by default to TrtllmAttention: By default, AttentionBackend.forward dispatches to TrtllmAttention, reading and updating paged KV cache blocks.
Piecewise CUDA Graph produces logits for Sampler: After replaying the Piecewise or generation CUDA Graph segments, the model forward outputs logits to the asynchronous Sampler.
TrtllmAttention executes CUDA kernels on NVIDIA GPU: TrtllmAttention launches its custom CUDA attention kernels on the GPU, writing in-place into preallocated buffers during Piecewise CUDA Graph execution.
Sampler samples next tokens on NVIDIA GPU: Sampling runs asynchronously on the GPU while the Overlap Scheduler processes the previous step's completed tokens on the CPU.
PyTorch is now the sole execution backend
Early releases of TensorRT-LLM required converting weights and running `trtllm-build` to compile a closed TensorRT `.engine` plan file ahead of time. TensorRT-LLM has since removed the TensorRT engine backend entirely: PyTorch is now the sole execution backend, HuggingFace checkpoints load directly into `tensorrt_llm.LLM`, and models are written in native PyTorch under `tensorrt_llm/_torch/models/`.
Inside each rank's `PyExecutor` worker, latency-critical control-plane pieces—the two-stage in-flight batching scheduler (`CapacityScheduler` and `MicroBatchScheduler`) and the paged `KVCacheManager`—run through C++ bindings (`cpp/tensorrt_llm/`), while the Overlap Scheduler launches step $n+1$ on the GPU before the CPU finishes post-processing step $n$.
How TensorRT-LLM's torch.compile backend differs from vLLM's
Both vLLM and TensorRT-LLM use PyTorch's TorchDynamo to capture the model graph and split it around attention for Piecewise CUDA Graphs, but what happens after Dynamo is different. vLLM hands each subgraph to TorchInductor to generate fused Triton kernels.
TensorRT-LLM does not use TorchInductor. Its custom `torch.compile` backend calls `aot_module_simplified` to lower the FX graph to ATen IR, runs custom pattern-matcher passes to fuse cross-module `AllReduce + Residual + RMSNorm` (+ FP8/FP4 quantization) into custom CUDA ops, strips `auto_functionalize` mutation copies in-place, inserts `trtllm.set_stream` / `record_event` / `wait_event` ops for multi-stream execution, and rewrites attention to write in-place into preallocated CUDA Graph buffers.
Every stage
LLM API / serve
application
Your Python code calling LLM.generate() or an OpenAI-compatible HTTP client calling trtllm-serve. The LLM class handles tokenization and detokenization.
The `LLM` class automatically manages essential pre and post-processing steps, including tokenization (encoding input prompts into numerical representations) and detokenization (decoding model outputs back into human-readable text).
A dedicated worker process per rank running an asynchronous background loop, with the Overlap Scheduler enabled by default to overlap CPU post-processing with GPU execution.
`PyExecutor` calls `ModelEngine.forward` with scheduled requests, runtime resources, and the current speculative decoding state; the engine builds `ScheduledInputs` and delegates to a `ScheduledModelRunner`.
The model defined in native PyTorch under tensorrt_llm/_torch/models/. PyTorch is the sole execution backend of TensorRT-LLM; HuggingFace checkpoints load directly without an engine build step.
Without Piecewise CUDA Graphs, context and mixed batches execute the PyTorch decoder eagerly while generation-only CUDA graphs are captured from the eager decoder.
With this option enabled, ineligible context and mixed forwards use the eager decoder. Ordinary CUDA graphs are also captured from the eager decoder and subsequently replayed as CUDA graphs.
Attention is wrapped as an opaque PyTorch custom op with a fake kernel and AttentionMetadata, writing in-place into preallocated tensors so Piecewise CUDA Graph segments keep stable addresses.
Lowers Torch IR via aot_module_simplified to ATen IR, fuses AllReduce + Residual + RMSNorm (+ FP8/FP4), strips auto_functionalize copies in-place, and schedules multi-stream ops without using TorchInductor.
The TRT-LLM backend, `TrtllmAttention`, serves as the default backend and supports all the features available in the Flashinfer backend while being further optimized for enhanced performance.
Context and mixed batches pad the token count and replay captured Piecewise CUDA Graph segments around eager attention; pure generation iterations replay generation-only CUDA Graphs.
We implement Piecewise CUDA Graph execution on top of torch.compile: non-capturable regions run in eager mode, while the rest of the model is captured and replayed as CUDA Graph segments.
Piecewise CUDA Graph only handles context-only and mixed context+generation iterations, while the generation-only CUDA Graph only handles pure generation iterations.
Pure generation steps replay captured CUDA Graphs padded to the nearest supported batch size, launching the entire sequence of decode kernels with one API call.
By capturing a sequence of CUDA operations as a single graph, the entire sequence can be launched with one API call, minimizing CPU-GPU synchronization and driver overhead.
Applies greedy, top-k, top-p, or beam-search sampling to the raw logits from ModelEngine asynchronously while the Overlap Scheduler processes the previous batch.
`Sampler`: Takes the raw outputs (logits) from the ModelEngine and applies appropriate sampling strategies (e.g., greedy, top-k, top-p, beam search) to generate the final output tokens.
The key strategy is to launch the GPU's work for the next step (n+1) immediately, without waiting for the CPU to finish processing the results of the current step (n).
Default (eager prefill + decode CUDA Graph) · any attention backend
Without torch.compile Piecewise CUDA Graphs, context and mixed batches run through the eager PyTorch decoder, launching custom ops and ATen kernels directly.
torch.compile + Piecewise CUDA Graph · any attention backend
TensorRT-LLM's custom backend lowers the FX graph to ATen IR, fuses AllReduce + Residual + RMSNorm, removes mutation-wrapper copies, and inserts multi-stream scheduling ops.
torch.compile + Piecewise CUDA Graph · any attention backend
Non-capturable attention is split out while the surrounding fused ATen/custom-op segments are captured per token-count bucket as Piecewise CUDA Graphs.
any compilation & graph mode · TrtllmAttention (default)
TrtllmAttention launches its custom CUDA attention kernels on the GPU, writing in-place into preallocated buffers during Piecewise CUDA Graph execution.
Documented execution paths for a causal LLM running on an NVIDIA GPU in TensorRT-LLM's PyTorch backend (the sole execution backend after the legacy TensorRT engine backend was removed), in either torch.compile + Piecewise CUDA Graph mode or default eager-prefill + generation CUDA Graph mode, using the default TrtllmAttention or FlashInferAttention backend. Multi-node tensor/pipeline/expert parallelism, DeepGEMM/MegaMoE barrier internals, speculative decoding, visual diffusion models (VisualGen), the experimental breakable CUDA Graph backend, and the removed legacy trtllm-build TensorRT engine pipeline are out of scope. The removed legacy TensorRT engine backend (`trtllm-build` / `LLM(backend="tensorrt")`) and closed NVIDIA TensorRT compiler internals are out of scope; TensorRT-LLM's migration guide states that PyTorch is now the sole execution backend.
How this page is checked
Every stage and step above quotes the source that supports it. The quotes were checked against the NVIDIA TensorRT-LLM documentation and repository on main (Developer Guide Architecture Overview, PyTorch Backend Architecture Overview, Scheduler, KV Cache Manager, Attention, Torch Compile & Prefill CUDA Graph, Quick Start Guide, and TensorRT Backend Removal Migration Guide) on 2026-11-11. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.