What Actually Happens When vLLM Serves a Request?

A request to vllm serve goes to the API server process, which tokenizes it and passes it to the engine core process. The engine core's scheduler batches it with other requests and sends the batch to a GPU worker, whose model runner calls the model: an ordinary PyTorch module. By default vLLM compiles that model with torch.compile before serving: Dynamo captures it, the graph is split at attention, and TorchInductor generates Triton kernels that are replayed through CUDA Graphs. With --enforce-eager, vLLM skips compilation and runs its own CUDA kernels and PyTorch's ATen ops directly. Attention always goes through an attention backend, FlashAttention by default on Ampere and Hopper GPUs, or Triton.

Showing the default torch.compile + CUDA Graphs mode with the FlashAttention backend: 14 stages, 14 steps.

Your program
Runtime
Framework core
Kernel
Compiler
Intermediate representation
Hardware

The selected path, step by step

Your app or client sends a request to API server: Your client sends an HTTP request, for example to the OpenAI-compatible chat completions endpoint.
API server passes tokens over ZMQ to Engine core: The API server tokenizes the prompt and hands the request to the engine core process over ZMQ sockets.
Engine core schedules each step on GPU worker: Every step the scheduler fills a token budget from running and waiting requests, allocates KV-cache blocks for them, and sends the batch to the GPU workers.
GPU worker runs the batch through Model runner: The worker's model runner prepares the input tensors for the batch.
Model runner calls forward() on PyTorch model: The model runner calls the model's forward pass on the batch.
PyTorch model calls attention through Attention op: Each attention layer calls one vLLM custom op, in both modes.
PyTorch model captured once at startup by TorchDynamo: Dynamo traces the model's forward function before vLLM serves anything; no request triggers a new compilation.
Attention op runs on FlashAttention: On Ampere and Hopper GPUs, FlashAttention is the first backend vLLM tries when none is selected.
TorchDynamo split at attention into Piecewise graph: The captured graph is cut at each attention call; attention stays a single opaque op between the pieces.
Piecewise graph compiled by TorchInductor: Each piece is compiled by TorchInductor, and the result is cached on disk for the next start.
TorchInductor generates Triton kernels: Under compilation vLLM turns its custom ops off by default and Inductor generates fused Triton kernels in their place.
Triton kernels recorded into CUDA Graphs: The model runner records the compiled kernels' launches as CUDA Graphs and replays them each step.
CUDA Graphs replayed on NVIDIA GPU: Replaying a graph launches its whole recorded sequence of kernels at once.
FlashAttention runs on NVIDIA GPU: Attention kernels run on the GPU; in the default compiled mode, single-token decode batches can include them in a full CUDA Graph.

Why vLLM is several processes

HTTP handling and tokenization run in the API server process; scheduling and KV-cache management run in the engine core process; each GPU has its own worker process. vLLM splits the work this way to separate concerns and maximize throughput; the processes talk over ZMQ sockets. Each step, the scheduler fills a token budget from both running and waiting requests, so there are no separate prefill and decode phases.

The KV cache is managed in fixed-size blocks. The scheduler's KV-cache manager builds the block tables, and the attention backend reads keys and values through them: this is PagedAttention, which today lives inside the attention backends rather than in a separate kernel.

Compiled before the first request

vLLM uses the same compiler stack as PyTorch's torch.compile, with one difference that matters for a server: every compilation finishes before vLLM serves any request, and the result is cached on disk, so no request triggers a recompilation.

The captured graph is split at each attention call. The pieces between attention calls are compiled by TorchInductor and captured as CUDA Graphs; by default, batches of single-token decode steps can run as one full CUDA Graph that includes attention. For a local, single-file engine built on hand-written kernels instead, compare how llama.cpp runs a model.

Every stage

Your app or client

application

Any program that sends requests to vLLM's OpenAI-compatible HTTP API and reads the streamed results.

The API server process handles HTTP requests (e.g., the OpenAI-compatible API), performs input processing (tokenization, multi-modal data loading), and streams results back to clients.
Architecture Overview (vLLM documentation) Project docs

API server

framework component

The process started by vllm serve. It handles HTTP, tokenizes inputs, and streams outputs back.

In the atlas: vLLM

This server can be started using the vllm serve command.
Architecture Overview (vLLM documentation) Project docs
The API server process handles HTTP requests (e.g., the OpenAI-compatible API), performs input processing (tokenization, multi-modal data loading), and streams results back to clients.
Architecture Overview (vLLM documentation) Project docs

Engine core

framework component

The process that runs the scheduler and manages the KV cache. Each step it picks which requests run and how many of their tokens.

In the atlas: vLLM

The engine core process runs the scheduler, manages KV cache, and coordinates model execution across GPU workers.
Architecture Overview (vLLM documentation) Project docs
This single loop handles chunked prefill, prefix caching and speculative decoding.
Architecture Overview (vLLM documentation) Project docs

GPU worker

runtime

One process per GPU. It loads the model weights, runs forward passes, and manages GPU memory.

In the atlas: vLLM

The worker process loads model weights, executes forward passes, and manages GPU memory.
Architecture Overview (vLLM documentation) Project docs

Model runner

framework component

Inside each worker: prepares the input tensors for the batch and captures the CUDA Graphs.

In the atlas: vLLM

Every worker has one model runner object, responsible for loading and running the model.
Architecture Overview (vLLM documentation) Project docs
Much of the model execution logic resides here, such as preparing input tensors and capturing cudagraphs.
Architecture Overview (vLLM documentation) Project docs

PyTorch model

framework component

The model itself, an ordinary PyTorch nn.Module (for example vLLM's Llama implementation).

In the atlas: PyTorch

Every model runner object has one model object, which is the actual torch.nn.Module instance.
Architecture Overview (vLLM documentation) Project docs

Eager CUDA kernels

kernel

Without compilation, layers that are vLLM custom ops run vLLM's own CUDA kernels, and the rest run as PyTorch ATen operations, one kernel launch at a time.

In the atlas: CUDA

it will automatically dispatch the forward method to the appropriate backend according to current_platform
CustomOp (vLLM documentation) Project docs
Most of these operations have both CPU and GPU implementations, to which the Tensor class will dynamically dispatch based on its type.
PyTorch C++ API: ATen (PyTorch documentation) Project docs

Attention op

framework component

vLLM wraps each attention call in one PyTorch custom op, so the compiler treats it as a single opaque operation.

In the atlas: vLLM

we wrap the whole attention operation into a PyTorch custom op torch.ops.vllm.unified_attention_with_output, so that Dynamo will not try to inspect any of the internal operations.
torch.compile integration (vLLM documentation) Project docs

TorchDynamo

compiler

PyTorch's graph capture. vLLM runs it on the model's forward function once, at startup.

In vLLM's V1 architecture, torch.compile is enabled by default and is a critical part of the framework.
torch.compile integration (vLLM documentation) Project docs
This is about the Python code compilation, i.e. graph capture by Dynamo.
torch.compile integration (vLLM documentation) Project docs

FlashAttention

kernel

The FlashAttention-based backend, vLLM's first choice for standard attention on Ampere and Hopper GPUs. It reads keys and values from the paged KV cache.

When no backend is explicitly selected, vLLM chooses the first compatible backend from these priority-ordered lists.
Attention Backend Feature Support (vLLM documentation) Project docs
Ampere/Hopper (SM 8.x-9.x): Priority Backend 1 FLASH_ATTN 2 FLASHINFER 3 TRITON_ATTN
Attention Backend Feature Support (vLLM documentation) Project docs

Triton attention

kernel

An attention backend written in Triton (TritonAttentionImpl). Selected with --attention-backend TRITON_ATTN, or automatically when the backends ahead of it do not fit the model.

In the atlas: Triton

Paged attention is now implemented directly inside vLLM's attention backends (for example TritonAttentionImpl in vllm/v1/attention/backends/triton_attn.py, or the FlashAttention-based backend) using block tables built by the scheduler's KV-cache manager
Paged Attention (vLLM documentation) Project docs
Ampere/Hopper (SM 8.x-9.x): Priority Backend 1 FLASH_ATTN 2 FLASHINFER 3 TRITON_ATTN
Attention Backend Feature Support (vLLM documentation) Project docs

Piecewise graph

intermediate representation

The captured computation graph, split at each attention call into pieces that are compiled one by one.

The computation graph is further split into pieces, by the splitting_ops (usually this is the attention operation).
torch.compile integration (vLLM documentation) Project docs

TorchInductor

compiler

PyTorch's compiler backend. It compiles each graph piece; vLLM caches the results on disk.

This means the first piece of computation graph (with shape None for symbolic shape) is compiled by Inductor
torch.compile integration (vLLM documentation) Project docs

Triton kernels

kernel

Fused GPU kernels that Inductor generates in Triton, including for the vLLM custom ops that compilation turns off.

In the atlas: Triton

In this case, Inductor generates (fused) Triton kernels for those disabled custom ops.
CustomOp (vLLM documentation) Project docs

CUDA Graphs

runtime

Recorded sequences of kernel launches that are replayed as one unit. By default: full graphs for decode batches, piecewise graphs (attention left outside) for the rest.

In the atlas: CUDA

FULL_AND_PIECEWISE — (default mode) full CUDA Graph for uniform decode, piecewise CUDA Graphs for others
CUDA Graphs (vLLM documentation) Project docs

NVIDIA GPU

hardware

The GPU that runs the kernels. This page covers Ampere and Hopper GPUs, where vLLM's attention priority list starts with FlashAttention.

Ampere/Hopper (SM 8.x-9.x): Priority Backend 1 FLASH_ATTN 2 FLASHINFER 3 TRITON_ATTN
Attention Backend Feature Support (vLLM documentation) Project docs

Every step

From	Step	To	When	What happens	Source
Your app or client	sends a request to	API server	any mode · any attention backend	Your client sends an HTTP request, for example to the OpenAI-compatible chat completions endpoint.	Project docs
API server	passes tokens over ZMQ to	Engine core	any mode · any attention backend	The API server tokenizes the prompt and hands the request to the engine core process over ZMQ sockets.	Project docs
Engine core	schedules each step on	GPU worker	any mode · any attention backend	Every step the scheduler fills a token budget from running and waiting requests, allocates KV-cache blocks for them, and sends the batch to the GPU workers.	Project docs
GPU worker	runs the batch through	Model runner	any mode · any attention backend	The worker's model runner prepares the input tensors for the batch.	Project docs
Model runner	calls forward() on	PyTorch model	any mode · any attention backend	The model runner calls the model's forward pass on the batch.	Project docs
PyTorch model	runs layers as	Eager CUDA kernels	--enforce-eager · any attention backend	With --enforce-eager there is no compilation or graph capture: each layer launches its kernels as Python reaches it. Startup is fastest; steady-state decoding is slower.	Project docs Project docs
PyTorch model	calls attention through	Attention op	any mode · any attention backend	Each attention layer calls one vLLM custom op, in both modes.	Project docs
PyTorch model	captured once at startup by	TorchDynamo	Default (compiled) · any attention backend	Dynamo traces the model's forward function before vLLM serves anything; no request triggers a new compilation.	Project docs Project docs
Attention op	runs on	FlashAttention	any mode · FlashAttention	On Ampere and Hopper GPUs, FlashAttention is the first backend vLLM tries when none is selected.	Project docs Project docs
Attention op	runs on	Triton attention	any mode · Triton	The Triton backend implements paged attention in Triton kernels, reading keys and values through the scheduler's block tables.	Project docs
TorchDynamo	split at attention into	Piecewise graph	Default (compiled) · any attention backend	The captured graph is cut at each attention call; attention stays a single opaque op between the pieces.	Project docs
Piecewise graph	compiled by	TorchInductor	Default (compiled) · any attention backend	Each piece is compiled by TorchInductor, and the result is cached on disk for the next start.	Project docs
TorchInductor	generates	Triton kernels	Default (compiled) · any attention backend	Under compilation vLLM turns its custom ops off by default and Inductor generates fused Triton kernels in their place.	Project docs
Triton kernels	recorded into	CUDA Graphs	Default (compiled) · any attention backend	The model runner records the compiled kernels' launches as CUDA Graphs and replays them each step.	Project docs Project docs
CUDA Graphs	replayed on	NVIDIA GPU	Default (compiled) · any attention backend	Replaying a graph launches its whole recorded sequence of kernels at once.	Project docs
Eager CUDA kernels	launch on	NVIDIA GPU	--enforce-eager · any attention backend	Each kernel is launched on the GPU individually.	Project docs
FlashAttention	runs on	NVIDIA GPU	any mode · FlashAttention	Attention kernels run on the GPU; in the default compiled mode, single-token decode batches can include them in a full CUDA Graph.	Project docs
Triton attention	runs on	NVIDIA GPU	any mode · Triton	Attention kernels run on the GPU; in the default compiled mode, single-token decode batches can include them in a full CUDA Graph.	Project docs

What this does not show

Documented paths for one model served with vllm serve on a single NVIDIA Ampere or Hopper GPU, in the default compiled mode or with --enforce-eager, using the FlashAttention or Triton attention backend. Tensor, pipeline and data parallelism, AMD, TPU, CPU and other platforms, FlashInfer and MLA backends, quantized GEMM kernels, speculative decoding, multimodal encoders and the experimental Rust frontend are out of scope.

How this page is checked

Every stage and step above quotes the source that supports it. The quotes were checked against the vLLM documentation (Architecture Overview, torch.compile integration, CUDA Graphs, CustomOp, Attention Backend Feature Support, Paged Attention, Optimization and Tuning), the vLLM README and the PyTorch ATen documentation on 2026-11-11. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.

What these tools are written in is in the main atlas: vLLM, PyTorch, CUDA, Triton. More execution paths: What Actually Happens When You Run JAX Code, How Does llama.cpp Run a Model on Your Hardware, What Actually Happens When You Run MLX on a Mac, How Does ONNX Runtime Execute a Model Across Hardware, What Actually Happens When You Run a PyTorch Model, or compare them all.

Explore the full graph →