What Actually Happens When SGLang Serves a Request?

A request to SGLang's HTTP server goes to the TokenizerManager in the same process, which tokenizes it and sends it over ZMQ to the Scheduler process. The scheduler matches the prompt against the radix prefix cache (RadixAttention), so tokens shared with earlier requests are not prefilled again, forms a batch and runs it through the ModelRunner. Decode steps replay CUDA graphs, and attention runs on the backend SGLang picks for your GPU: FlashInfer on Ampere and Ada, FlashAttention 3 on Hopper, TRT-LLM MHA on Blackwell. Output tokens go to the DetokenizerManager process, which returns text to be streamed back.
GPU generation

Showing an SGLang request on a Hopper GPU (FlashAttention 3): 10 stages, 11 steps.

Your app or client sends a request to HTTP server HTTP server hands the request to TokenizerManager TokenizerManager sends tokens over ZMQ to Scheduler Scheduler matches prefixes in Radix prefix cache Scheduler forwards each batch to ModelRunner ModelRunner replays decode steps as Decode CUDA graphs ModelRunner computes attention with FlashInfer ModelRunner computes attention with FlashAttention 3 ModelRunner computes attention with TRT-LLM MHA FlashInfer runs on NVIDIA GPU FlashAttention 3 runs on NVIDIA GPU TRT-LLM MHA runs on NVIDIA GPU Decode CUDA graphs replayed on NVIDIA GPU Scheduler sends output tokens to DetokenizerManager DetokenizerManager returns text to TokenizerManager Your app or client application HTTP server framework component TokenizerManager framework component Scheduler runtime DetokenizerManager framework component Radix prefix cache framework component ModelRunner framework component Decode CUDA graphs runtime FlashInfer kernel FlashAttention 3 kernel TRT-LLM MHA kernel NVIDIA GPU hardware

The selected path, step by step

  1. Your app or client sends a request to HTTP server: Your client sends an HTTP request to the server.
  2. HTTP server hands the request to TokenizerManager: The HTTP server and the TokenizerManager share the main process.
  3. TokenizerManager sends tokens over ZMQ to Scheduler: Tokenized requests go to the scheduler process over ZMQ.
  4. Scheduler matches prefixes in Radix prefix cache: The scheduler matches each request against the cached prefixes, so the KV cache of tokens shared with earlier requests is reused rather than computed again.
  5. Scheduler forwards each batch to ModelRunner: The scheduler forms a batch and runs it through the model.
  6. ModelRunner replays decode steps as Decode CUDA graphs: Decode batches replay captured CUDA graphs, one per batch shape by default.
  7. ModelRunner computes attention with FlashAttention 3: With no --attention-backend set, Hopper GPUs use FlashAttention 3.
  8. FlashAttention 3 runs on NVIDIA GPU: The attention kernels run on the GPU.
  9. Decode CUDA graphs replayed on NVIDIA GPU: Replaying a graph launches the step's kernels together.
  10. Scheduler sends output tokens to DetokenizerManager: Output tokens go to the detokenizer process.
  11. DetokenizerManager returns text to TokenizerManager: Decoded text goes back to the TokenizerManager, which returns it to the client.

Why prefix caching matters

When many requests share the same prefix, the KV cache for that prefix is identical. SGLang keeps the KV cache of those tokens in a radix tree, where each node covers a span of tokens and a path from the root is a request's prefix. A new request that shares the path skips computing it, and the shared nodes are stored once.

How this compares with vLLM

Both engines split serving into processes: SGLang uses a TokenizerManager, a scheduler process and a DetokenizerManager talking over ZMQ; vLLM uses an API server, an engine core and GPU workers. Both replay CUDA graphs and choose an attention backend by GPU. Which one is faster depends on the workload and version, so this page does not compare speed.

Every stage

Scheduler

runtime

A separate process that receives requests, forms batches, runs them through the model and sends the output tokens on.

In the atlas: SGLang

DetokenizerManager

framework component

A separate process that turns output token ids back into text and returns it to the TokenizerManager.

In the atlas: SGLang

Radix prefix cache

framework component

RadixAttention's radix tree over the KV cache: each node holds the KV of a span of tokens, so requests that share a prefix reuse it instead of recomputing it.

In the atlas: SGLang

FlashInfer

kernel

Default attention backend for standard (MHA) models on Ampere, Ada and other non-Hopper, non-Blackwell NVIDIA GPUs, when available.

FlashAttention 3

kernel

Default attention backend on Hopper GPUs (H100, H200) with CUDA 12.3 or newer, for supported models.

TRT-LLM MHA

kernel

Default attention backend on Blackwell GPUs (B200), unless speculative decoding uses topk > 1.

NVIDIA GPU

hardware

The GPU that runs the batch.

Every step

FromStepToWhenWhat happensSource
Your app or clientsends a request toHTTP serverany gpu generationYour client sends an HTTP request to the server.Source repository
HTTP serverhands the request toTokenizerManagerany gpu generationThe HTTP server and the TokenizerManager share the main process.Source repository
TokenizerManagersends tokens over ZMQ toSchedulerany gpu generationTokenized requests go to the scheduler process over ZMQ.Source repository
Source repository
Schedulermatches prefixes inRadix prefix cacheany gpu generationThe scheduler matches each request against the cached prefixes, so the KV cache of tokens shared with earlier requests is reused rather than computed again.Source repository
Source repository
Schedulerforwards each batch toModelRunnerany gpu generationThe scheduler forms a batch and runs it through the model.Source repository
Source repository
ModelRunnerreplays decode steps asDecode CUDA graphsany gpu generationDecode batches replay captured CUDA graphs, one per batch shape by default.Source repository
Source repository
ModelRunnercomputes attention withFlashInferAmpere / AdaWith no --attention-backend set, Ampere and Ada GPUs use FlashInfer when it is available.Source repository
Source repository
ModelRunnercomputes attention withFlashAttention 3Hopper (H100)With no --attention-backend set, Hopper GPUs use FlashAttention 3.Source repository
Source repository
ModelRunnercomputes attention withTRT-LLM MHABlackwell (B200)With no --attention-backend set, Blackwell GPUs use TensorRT-LLM's MHA kernels.Source repository
Source repository
FlashInferruns onNVIDIA GPUAmpere / AdaThe attention kernels run on the GPU.Source repository
FlashAttention 3runs onNVIDIA GPUHopper (H100)The attention kernels run on the GPU.Source repository
TRT-LLM MHAruns onNVIDIA GPUBlackwell (B200)The attention kernels run on the GPU.Source repository
Decode CUDA graphsreplayed onNVIDIA GPUany gpu generationReplaying a graph launches the step's kernels together.Source repository
Schedulersends output tokens toDetokenizerManagerany gpu generationOutput tokens go to the detokenizer process.Source repository
DetokenizerManagerreturns text toTokenizerManagerany gpu generationDecoded text goes back to the TokenizerManager, which returns it to the client.Source repository

What this does not show

Documented paths for one standard (MHA) model such as Llama or Qwen served on a single NVIDIA GPU with default settings, by GPU generation. MLA models, AMD and other hardware, tensor and data parallelism, speculative decoding, HiCache's host and remote tiers, the optional torch.compile mode and SGLang's frontend language are out of scope.

How this page is checked

Every stage and step above quotes the source that supports it. The quotes were checked against the SGLang source (engine, HTTP server, scheduler, model runner, radix cache and decode CUDA graph runner) and its documentation sources for HiCache design and attention backends on 2026-11-12. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.

See this path in the graph explorer →

What these tools are written in is in the main atlas: SGLang, CUDA. More execution paths: How Does a Rust LLM Generate Tokens with Candle, What Actually Happens When You Run JAX Code, What Runs Underneath a Keras 3 Model, How Does llama.cpp Run a Model on Your Hardware, How Does MLC LLM Turn a Compiled Model into Generated Tokens, What Actually Happens When You Run MLX on a Mac, What Actually Runs When You Send Ollama a Prompt, How Does ONNX Runtime Execute a Model Across Hardware, What Actually Happens When You Run a PyTorch Model, How Does TensorFlow Run Python on a CPU or GPU, What Actually Happens When TensorRT-LLM Runs a Model, What Actually Happens When You Run a Hugging Face Pipeline, What Actually Happens When vLLM Serves a Request, or compare them all.

Explore the full graph →