Your app or client
application
Any program that sends requests to SGLang's HTTP API.
This file implements HTTP APIs for the inference engine via fastapi.
python/sglang/srt/entrypoints/http_server.py (sgl-project/sglang) Source repository
Showing an SGLang request on a Hopper GPU (FlashAttention 3): 10 stages, 11 steps.
When many requests share the same prefix, the KV cache for that prefix is identical. SGLang keeps the KV cache of those tokens in a radix tree, where each node covers a span of tokens and a path from the root is a request's prefix. A new request that shares the path skips computing it, and the shared nodes are stored once.
Both engines split serving into processes: SGLang uses a TokenizerManager, a scheduler process and a DetokenizerManager talking over ZMQ; vLLM uses an API server, an engine core and GPU workers. Both replay CUDA graphs and choose an attention backend by GPU. Which one is faster depends on the workload and version, so this page does not compare speed.
application
Any program that sends requests to SGLang's HTTP API.
This file implements HTTP APIs for the inference engine via fastapi.python/sglang/srt/entrypoints/http_server.py (sgl-project/sglang) Source repository
framework component
SGLang's FastAPI server, running in the main process alongside the engine and the TokenizerManager.
In the atlas: SGLang
This file implements HTTP APIs for the inference engine via fastapi.python/sglang/srt/entrypoints/http_server.py (sgl-project/sglang) Source repository
The HTTP server, Engine, and TokenizerManager all run in the main process.python/sglang/srt/entrypoints/engine.py (sgl-project/sglang) Source repository
framework component
Tokenizes each request and hands it to the scheduler process.
In the atlas: SGLang
TokenizerManager: Tokenizes the requests and sends them to the scheduler.python/sglang/srt/entrypoints/engine.py (sgl-project/sglang) Source repository
runtime
A separate process that receives requests, forms batches, runs them through the model and sends the output tokens on.
In the atlas: SGLang
Scheduler (subprocess): Receives requests from the Tokenizer Manager, schedules batches, forwards them, and sends the output tokens to the Detokenizer Manager.python/sglang/srt/entrypoints/engine.py (sgl-project/sglang) Source repository
A scheduler that manages a tensor parallel GPU worker.python/sglang/srt/managers/scheduler.py (sgl-project/sglang) Source repository
framework component
A separate process that turns output token ids back into text and returns it to the TokenizerManager.
In the atlas: SGLang
DetokenizerManager (subprocess): Detokenizes the output tokens and sends the result back to the Tokenizer Manager.python/sglang/srt/entrypoints/engine.py (sgl-project/sglang) Source repository
framework component
RadixAttention's radix tree over the KV cache: each node holds the KV of a span of tokens, so requests that share a prefix reuse it instead of recomputing it.
In the atlas: SGLang
The radix tree data structure for managing the KV cache.python/sglang/srt/mem_cache/radix_cache.py (sgl-project/sglang) Source repository
In RadixAttention, each node of the RadixTree corresponds to the KV cache of a consecutive span of tokens in GPU memory.HiCache design (SGLang documentation source) Source repository
framework component
Runs the model's forward pass for each batch.
In the atlas: SGLang
ModelRunner runs the forward passes of the models.python/sglang/srt/model_executor/model_runner.py (sgl-project/sglang) Source repository
runtime
Decode steps run as captured CUDA graphs; the default backend keeps one full graph per batch shape.
In the atlas: CUDA
DecodeCudaGraphRunner — runs DECODE / TARGET_VERIFY / DLLM_EXTEND underpython/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py (sgl-project/sglang) Source repository
default, FullCudaGraphBackend: onepython/sglang/srt/model_executor/runner/decode_cuda_graph_runner.py (sgl-project/sglang) Source repository
kernel
Default attention backend for standard (MHA) models on Ampere, Ada and other non-Hopper, non-Blackwell NVIDIA GPUs, when available.
Defaults to flashinfer if available; otherwise falls back to triton.Attention backend (SGLang documentation source) Source repository
kernel
Default attention backend on Hopper GPUs (H100, H200) with CUDA 12.3 or newer, for supported models.
Defaults to fa3 if using CUDA 12.3+ and the model configuration is supported.Attention backend (SGLang documentation source) Source repository
kernel
Default attention backend on Blackwell GPUs (B200), unless speculative decoding uses topk > 1.
Defaults to trtllm_mha, unless using speculative decoding with topk > 1.Attention backend (SGLang documentation source) Source repository
hardware
The GPU that runs the batch.
If the --attention-backend argument is not specified, SGLang automatically selects the best backend based on the hardware (CUDA) and model architecture.Attention backend (SGLang documentation source) Source repository
| From | Step | To | When | What happens | Source |
|---|---|---|---|---|---|
| Your app or client | sends a request to | HTTP server | any gpu generation | Your client sends an HTTP request to the server. | Source repository |
| HTTP server | hands the request to | TokenizerManager | any gpu generation | The HTTP server and the TokenizerManager share the main process. | Source repository |
| TokenizerManager | sends tokens over ZMQ to | Scheduler | any gpu generation | Tokenized requests go to the scheduler process over ZMQ. | Source repository Source repository |
| Scheduler | matches prefixes in | Radix prefix cache | any gpu generation | The scheduler matches each request against the cached prefixes, so the KV cache of tokens shared with earlier requests is reused rather than computed again. | Source repository Source repository |
| Scheduler | forwards each batch to | ModelRunner | any gpu generation | The scheduler forms a batch and runs it through the model. | Source repository Source repository |
| ModelRunner | replays decode steps as | Decode CUDA graphs | any gpu generation | Decode batches replay captured CUDA graphs, one per batch shape by default. | Source repository Source repository |
| ModelRunner | computes attention with | FlashInfer | Ampere / Ada | With no --attention-backend set, Ampere and Ada GPUs use FlashInfer when it is available. | Source repository Source repository |
| ModelRunner | computes attention with | FlashAttention 3 | Hopper (H100) | With no --attention-backend set, Hopper GPUs use FlashAttention 3. | Source repository Source repository |
| ModelRunner | computes attention with | TRT-LLM MHA | Blackwell (B200) | With no --attention-backend set, Blackwell GPUs use TensorRT-LLM's MHA kernels. | Source repository Source repository |
| FlashInfer | runs on | NVIDIA GPU | Ampere / Ada | The attention kernels run on the GPU. | Source repository |
| FlashAttention 3 | runs on | NVIDIA GPU | Hopper (H100) | The attention kernels run on the GPU. | Source repository |
| TRT-LLM MHA | runs on | NVIDIA GPU | Blackwell (B200) | The attention kernels run on the GPU. | Source repository |
| Decode CUDA graphs | replayed on | NVIDIA GPU | any gpu generation | Replaying a graph launches the step's kernels together. | Source repository |
| Scheduler | sends output tokens to | DetokenizerManager | any gpu generation | Output tokens go to the detokenizer process. | Source repository |
| DetokenizerManager | returns text to | TokenizerManager | any gpu generation | Decoded text goes back to the TokenizerManager, which returns it to the client. | Source repository |
Documented paths for one standard (MHA) model such as Llama or Qwen served on a single NVIDIA GPU with default settings, by GPU generation. MLA models, AMD and other hardware, tensor and data parallelism, speculative decoding, HiCache's host and remote tiers, the optional torch.compile mode and SGLang's frontend language are out of scope.
Every stage and step above quotes the source that supports it. The quotes were checked against the SGLang source (engine, HTTP server, scheduler, model runner, radix cache and decode CUDA graph runner) and its documentation sources for HiCache design and attention backends on 2026-11-12. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.
See this path in the graph explorer →
What these tools are written in is in the main atlas: SGLang, CUDA. More execution paths: How Does a Rust LLM Generate Tokens with Candle, What Actually Happens When You Run JAX Code, What Runs Underneath a Keras 3 Model, How Does llama.cpp Run a Model on Your Hardware, How Does MLC LLM Turn a Compiled Model into Generated Tokens, What Actually Happens When You Run MLX on a Mac, What Actually Runs When You Send Ollama a Prompt, How Does ONNX Runtime Execute a Model Across Hardware, What Actually Happens When You Run a PyTorch Model, How Does TensorFlow Run Python on a CPU or GPU, What Actually Happens When TensorRT-LLM Runs a Model, What Actually Happens When You Run a Hugging Face Pipeline, What Actually Happens When vLLM Serves a Request, or compare them all.
Explore the full graph →