What Actually Happens When TensorRT-LLM Runs a Model?

A call to LLM.generate() or trtllm-serve tokenizes the prompt and submits it to a per-rank PyExecutor worker running an asynchronous loop with the Overlap Scheduler. Each step, C++-bound CapacityScheduler and MicroBatchScheduler pick active context and generation requests while the C++-bound KVCacheManager allocates paged KV-cache blocks. ModelEngine delegates the batch to a native PyTorch model (PyTorch is now TensorRT-LLM's sole execution backend; `trtllm-build` engines were removed). With Piecewise CUDA Graph enabled, TorchDynamo traces the model into Torch IR and TensorRT-LLM's custom torch.compile backend lowers it via aot_module_simplified to ATen IR—fusing AllReduce + RMSNorm and scheduling multiple CUDA streams without using TorchInductor—then replays Piecewise CUDA Graphs around in-place TrtllmAttention or FlashInferAttention kernels.
Compilation & graph mode
Attention backend

Showing torch.compile with Piecewise CUDA Graphs (prefill_cuda_graph_backend: piecewise) with the default TrtllmAttention backend: 13 stages, 13 steps.

LLM API / serve tokenizes and submits request to PyExecutor worker PyExecutor worker schedules active requests via C++ batch scheduler C++ batch scheduler checks capacity and reserves blocks in C++ KVCacheManager C++ KVCacheManager passes scheduled batch & KV buffers to ModelEngine & runner ModelEngine & runner runs forward pass on PyTorch model PyTorch model executes context/mixed batches via Eager prefill ops PyTorch model invokes attention through Attention custom op PyTorch model traced in fullgraph mode by TorchDynamo (FX) TorchDynamo (FX) lowered via aot_module_simplified to TRT-LLM ATen IR TRT-LLM ATen IR partitioned at attention into Piecewise CUDA Graph Eager prefill ops captures decode steps as Decode CUDA Graph Attention custom op dispatches by default to TrtllmAttention Attention custom op dispatches when selected to FlashInferAttention Piecewise CUDA Graph produces logits for Sampler Decode CUDA Graph produces logits for Sampler TrtllmAttention executes CUDA kernels on NVIDIA GPU FlashInferAttention executes CUDA kernels on NVIDIA GPU Sampler samples next tokens on NVIDIA GPU LLM API / serve application PyExecutor worker runtime C++ batch scheduler framework component C++ KVCacheManager framework component ModelEngine & runner framework component PyTorch model framework component Eager prefill ops kernel Attention custom op framework component TorchDynamo (FX) compiler TRT-LLM ATen IR compiler TrtllmAttention kernel FlashInferAttention kernel Piecewise CUDA Graph runtime Decode CUDA Graph runtime Sampler framework component NVIDIA GPU hardware

The selected path, step by step

  1. LLM API / serve tokenizes and submits request to PyExecutor worker: LLM.generate() or trtllm-serve tokenizes the prompt and queues the request for the rank's PyExecutor worker.
  2. PyExecutor worker schedules active requests via C++ batch scheduler: At each iteration of its background loop, PyExecutor invokes SimpleScheduler (combining CapacityScheduler and MicroBatchScheduler).
  3. C++ batch scheduler checks capacity and reserves blocks in C++ KVCacheManager: CapacityScheduler queries KVCacheManager for available blocks and PyExecutor calls prepare_resources to allocate paged KV-cache blocks for the batch.
  4. C++ KVCacheManager passes scheduled batch & KV buffers to ModelEngine & runner: PyExecutor invokes ModelEngine.forward with the scheduled context and generation requests and their allocated KV-cache resources.
  5. ModelEngine & runner runs forward pass on PyTorch model: ScheduledModelRunner prepares the input tensors and executes the PyTorch model's forward pass.
  6. PyTorch model invokes attention through Attention custom op: Each transformer layer prepares AttentionMetadata and calls the wrapped attention custom op.
  7. PyTorch model traced in fullgraph mode by TorchDynamo (FX): Enabling prefill_cuda_graph_backend: piecewise enables torch.compile, which traces the decoder model with TorchDynamo into a Torch IR FX graph.
  8. TorchDynamo (FX) lowered via aot_module_simplified to TRT-LLM ATen IR: TensorRT-LLM's custom backend lowers the FX graph to ATen IR, fuses AllReduce + Residual + RMSNorm, removes mutation-wrapper copies, and inserts multi-stream scheduling ops.
  9. TRT-LLM ATen IR partitioned at attention into Piecewise CUDA Graph: Non-capturable attention is split out while the surrounding fused ATen/custom-op segments are captured per token-count bucket as Piecewise CUDA Graphs.
  10. Attention custom op dispatches by default to TrtllmAttention: By default, AttentionBackend.forward dispatches to TrtllmAttention, reading and updating paged KV cache blocks.
  11. Piecewise CUDA Graph produces logits for Sampler: After replaying the Piecewise or generation CUDA Graph segments, the model forward outputs logits to the asynchronous Sampler.
  12. TrtllmAttention executes CUDA kernels on NVIDIA GPU: TrtllmAttention launches its custom CUDA attention kernels on the GPU, writing in-place into preallocated buffers during Piecewise CUDA Graph execution.
  13. Sampler samples next tokens on NVIDIA GPU: Sampling runs asynchronously on the GPU while the Overlap Scheduler processes the previous step's completed tokens on the CPU.

PyTorch is now the sole execution backend

Early releases of TensorRT-LLM required converting weights and running `trtllm-build` to compile a closed TensorRT `.engine` plan file ahead of time. TensorRT-LLM has since removed the TensorRT engine backend entirely: PyTorch is now the sole execution backend, HuggingFace checkpoints load directly into `tensorrt_llm.LLM`, and models are written in native PyTorch under `tensorrt_llm/_torch/models/`.

Inside each rank's `PyExecutor` worker, latency-critical control-plane pieces—the two-stage in-flight batching scheduler (`CapacityScheduler` and `MicroBatchScheduler`) and the paged `KVCacheManager`—run through C++ bindings (`cpp/tensorrt_llm/`), while the Overlap Scheduler launches step $n+1$ on the GPU before the CPU finishes post-processing step $n$.

How TensorRT-LLM's torch.compile backend differs from vLLM's

Both vLLM and TensorRT-LLM use PyTorch's TorchDynamo to capture the model graph and split it around attention for Piecewise CUDA Graphs, but what happens after Dynamo is different. vLLM hands each subgraph to TorchInductor to generate fused Triton kernels.

TensorRT-LLM does not use TorchInductor. Its custom `torch.compile` backend calls `aot_module_simplified` to lower the FX graph to ATen IR, runs custom pattern-matcher passes to fuse cross-module `AllReduce + Residual + RMSNorm` (+ FP8/FP4 quantization) into custom CUDA ops, strips `auto_functionalize` mutation copies in-place, inserts `trtllm.set_stream` / `record_event` / `wait_event` ops for multi-stream execution, and rewrites attention to write in-place into preallocated CUDA Graph buffers.

Every stage

LLM API / serve

application

Your Python code calling LLM.generate() or an OpenAI-compatible HTTP client calling trtllm-serve. The LLM class handles tokenization and detokenization.

In the atlas: Python

PyExecutor worker

runtime

A dedicated worker process per rank running an asynchronous background loop, with the Overlap Scheduler enabled by default to overlap CPU post-processing with GPU execution.

In the atlas: TensorRT-LLM

C++ batch scheduler

framework component

Two-stage C++-bound scheduler (CapacityScheduler and MicroBatchScheduler) that dynamically selects context and generation requests at every step.

In the atlas: C++

C++ KVCacheManager

framework component

C++-bound resource manager that allocates, updates, and recycles paged KV-cache blocks for the scheduled context and generation requests.

In the atlas: C++

ModelEngine & runner

framework component

Loads the model, builds ScheduledInputs for the current step, and delegates input preparation, CUDA graphs, and execution to ScheduledModelRunner.

In the atlas: TensorRT-LLM

PyTorch model

framework component

The model defined in native PyTorch under tensorrt_llm/_torch/models/. PyTorch is the sole execution backend of TensorRT-LLM; HuggingFace checkpoints load directly without an engine build step.

In the atlas: PyTorch

Eager prefill ops

kernel

Without Piecewise CUDA Graphs, context and mixed batches execute the PyTorch decoder eagerly while generation-only CUDA graphs are captured from the eager decoder.

In the atlas: CUDA

Attention custom op

framework component

Attention is wrapped as an opaque PyTorch custom op with a fake kernel and AttentionMetadata, writing in-place into preallocated tensors so Piecewise CUDA Graph segments keep stable addresses.

In the atlas: TensorRT-LLM

TorchDynamo (FX)

compiler

Traces the PyTorch model in fullgraph mode into a Torch IR FX graph, preserving tensor names so the compiler can track the dynamic token-count SymInt.

In the atlas: PyTorch

TRT-LLM ATen IR

compiler

Lowers Torch IR via aot_module_simplified to ATen IR, fuses AllReduce + Residual + RMSNorm (+ FP8/FP4), strips auto_functionalize copies in-place, and schedules multi-stream ops without using TorchInductor.

In the atlas: TensorRT-LLM

TrtllmAttention

kernel

The default production attention backend (TrtllmAttention): custom CUDA kernels supporting inflight batching, paged KV cache, fused QKV input, and FP8 output.

In the atlas: CUDA

FlashInferAttention

kernel

Optional FlashInfer attention backend (selected with attn_backend="flashinfer") supporting inflight batching, paged KV cache, FP8 quantization, and RoPE fusion.

In the atlas: CUDA

Piecewise CUDA Graph

runtime

Context and mixed batches pad the token count and replay captured Piecewise CUDA Graph segments around eager attention; pure generation iterations replay generation-only CUDA Graphs.

In the atlas: CUDA

Decode CUDA Graph

runtime

Pure generation steps replay captured CUDA Graphs padded to the nearest supported batch size, launching the entire sequence of decode kernels with one API call.

In the atlas: CUDA

Sampler

framework component

Applies greedy, top-k, top-p, or beam-search sampling to the raw logits from ModelEngine asynchronously while the Overlap Scheduler processes the previous batch.

In the atlas: TensorRT-LLM

NVIDIA GPU

hardware

Executes the replayed CUDA Graph segments, custom CUDA kernels (attention, GEMM, MoE, AllReduce), and asynchronous sampling on NVIDIA GPUs.

Every step

FromStepToWhenWhat happensSource
LLM API / servetokenizes and submits request toPyExecutor workerany compilation & graph mode · any attention backendLLM.generate() or trtllm-serve tokenizes the prompt and queues the request for the rank's PyExecutor worker.Source repository
Source repository
PyExecutor workerschedules active requests viaC++ batch schedulerany compilation & graph mode · any attention backendAt each iteration of its background loop, PyExecutor invokes SimpleScheduler (combining CapacityScheduler and MicroBatchScheduler).Source repository
Source repository
C++ batch schedulerchecks capacity and reserves blocks inC++ KVCacheManagerany compilation & graph mode · any attention backendCapacityScheduler queries KVCacheManager for available blocks and PyExecutor calls prepare_resources to allocate paged KV-cache blocks for the batch.Source repository
Source repository
C++ KVCacheManagerpasses scheduled batch & KV buffers toModelEngine & runnerany compilation & graph mode · any attention backendPyExecutor invokes ModelEngine.forward with the scheduled context and generation requests and their allocated KV-cache resources.Source repository
ModelEngine & runnerruns forward pass onPyTorch modelany compilation & graph mode · any attention backendScheduledModelRunner prepares the input tensors and executes the PyTorch model's forward pass.Source repository
Source repository
PyTorch modelexecutes context/mixed batches viaEager prefill opsDefault (eager prefill + decode CUDA Graph) · any attention backendWithout torch.compile Piecewise CUDA Graphs, context and mixed batches run through the eager PyTorch decoder, launching custom ops and ATen kernels directly.Source repository
PyTorch modelinvokes attention throughAttention custom opany compilation & graph mode · any attention backendEach transformer layer prepares AttentionMetadata and calls the wrapped attention custom op.Source repository
PyTorch modeltraced in fullgraph mode byTorchDynamo (FX)torch.compile + Piecewise CUDA Graph · any attention backendEnabling prefill_cuda_graph_backend: piecewise enables torch.compile, which traces the decoder model with TorchDynamo into a Torch IR FX graph.Source repository
TorchDynamo (FX)lowered via aot_module_simplified toTRT-LLM ATen IRtorch.compile + Piecewise CUDA Graph · any attention backendTensorRT-LLM's custom backend lowers the FX graph to ATen IR, fuses AllReduce + Residual + RMSNorm, removes mutation-wrapper copies, and inserts multi-stream scheduling ops.Source repository
Source repository
TRT-LLM ATen IRpartitioned at attention intoPiecewise CUDA Graphtorch.compile + Piecewise CUDA Graph · any attention backendNon-capturable attention is split out while the surrounding fused ATen/custom-op segments are captured per token-count bucket as Piecewise CUDA Graphs.Source repository
Eager prefill opscaptures decode steps asDecode CUDA GraphDefault (eager prefill + decode CUDA Graph) · any attention backendPure generation steps are captured from the eager decoder as generation-only CUDA Graphs with batch-size padding.Source repository
Source repository
Attention custom opdispatches by default toTrtllmAttentionany compilation & graph mode · TrtllmAttention (default)By default, AttentionBackend.forward dispatches to TrtllmAttention, reading and updating paged KV cache blocks.Source repository
Attention custom opdispatches when selected toFlashInferAttentionany compilation & graph mode · FlashInferAttentionWhen configured with attn_backend="flashinfer", AttentionBackend.forward calls FlashInferAttention's cached prefill or decode plan.Source repository
Source repository
Piecewise CUDA Graphproduces logits forSamplertorch.compile + Piecewise CUDA Graph · any attention backendAfter replaying the Piecewise or generation CUDA Graph segments, the model forward outputs logits to the asynchronous Sampler.Source repository
Decode CUDA Graphproduces logits forSamplerDefault (eager prefill + decode CUDA Graph) · any attention backendThe eager prefill or replayed generation CUDA Graph outputs logits to the asynchronous Sampler.Source repository
TrtllmAttentionexecutes CUDA kernels onNVIDIA GPUany compilation & graph mode · TrtllmAttention (default)TrtllmAttention launches its custom CUDA attention kernels on the GPU, writing in-place into preallocated buffers during Piecewise CUDA Graph execution.Source repository
FlashInferAttentionexecutes CUDA kernels onNVIDIA GPUany compilation & graph mode · FlashInferAttentionFlashInferAttention launches its paged prefill and decode CUDA kernels on the GPU.Source repository
Samplersamples next tokens onNVIDIA GPUany compilation & graph mode · any attention backendSampling runs asynchronously on the GPU while the Overlap Scheduler processes the previous step's completed tokens on the CPU.Source repository

What this does not show

Documented execution paths for a causal LLM running on an NVIDIA GPU in TensorRT-LLM's PyTorch backend (the sole execution backend after the legacy TensorRT engine backend was removed), in either torch.compile + Piecewise CUDA Graph mode or default eager-prefill + generation CUDA Graph mode, using the default TrtllmAttention or FlashInferAttention backend. Multi-node tensor/pipeline/expert parallelism, DeepGEMM/MegaMoE barrier internals, speculative decoding, visual diffusion models (VisualGen), the experimental breakable CUDA Graph backend, and the removed legacy trtllm-build TensorRT engine pipeline are out of scope. The removed legacy TensorRT engine backend (`trtllm-build` / `LLM(backend="tensorrt")`) and closed NVIDIA TensorRT compiler internals are out of scope; TensorRT-LLM's migration guide states that PyTorch is now the sole execution backend.

How this page is checked

Every stage and step above quotes the source that supports it. The quotes were checked against the NVIDIA TensorRT-LLM documentation and repository on main (Developer Guide Architecture Overview, PyTorch Backend Architecture Overview, Scheduler, KV Cache Manager, Attention, Torch Compile & Prefill CUDA Graph, Quick Start Guide, and TensorRT Backend Removal Migration Guide) on 2026-11-11. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.

See this path in the graph explorer →

What these tools are written in is in the main atlas: Python, TensorRT-LLM, C++, PyTorch, CUDA. More execution paths: How Does a Rust LLM Generate Tokens with Candle, What Actually Happens When You Run JAX Code, What Runs Underneath a Keras 3 Model, How Does llama.cpp Run a Model on Your Hardware, How Does MLC LLM Turn a Compiled Model into Generated Tokens, What Actually Happens When You Run MLX on a Mac, What Actually Runs When You Send Ollama a Prompt, How Does ONNX Runtime Execute a Model Across Hardware, What Actually Happens When You Run a PyTorch Model, What Actually Happens When SGLang Serves a Request, How Does TensorFlow Run Python on a CPU or GPU, What Actually Happens When You Run a Hugging Face Pipeline, What Actually Happens When vLLM Serves a Request, or compare them all.

Explore the full graph →