How Does MLC LLM Turn a Compiled Model into Generated Tokens?

MLC LLM needs two different artifacts: converted model weights and a target-compatible model library containing inference logic. A first run can use Apache TVM to compile that library; later runs can reuse matching cached artifacts. The native runtime tokenizes the prompt, embeds its token IDs and performs prefill to produce next-token logits. A sampler selects a token. Subsequent decode calls process the last committed token while the KV cache retains earlier attention keys and values. That loop produces more tokens; detokenization converts their IDs back into text. GPU execution and Python compilation are distinct parts of the system.
Startup

Showing a first run with compilation for a supported single-GPU MLC model: 12 stages, 15 steps.

MLC chat request obtains model parameters Converted weights MLC chat request compiles when needed TVM compilation TVM compilation generates inference logic Native model library MLC chat request reuses matching library Native model library Native model library is loaded into Native chat runtime Converted weights supply model parameters Native chat runtime Native chat runtime encodes prompt text Prompt tokenization Prompt tokenization embeds prompt IDs Prompt prefill Native chat runtime maintains sequence state Attention KV cache Attention KV cache provides sequence state Prompt prefill Prompt prefill returns next-token logits Token sampling Token sampling commits the next token Next-token decode Next-token decode returns new logits Token sampling Token sampling decodes selected IDs Detokenized output Prompt prefill executes compiled model on Selected local GPU Next-token decode executes compiled model on Selected local GPU MLC chat request application Converted weights model format TVM compilation compiler Native model library machine code Native chat runtime runtime Prompt tokenization framework component Attention KV cache framework component Prompt prefill framework component Next-token decode framework component Token sampling framework component Detokenized output application Selected local GPU hardware

The selected path, step by step

  1. MLC chat request obtains model parameters Converted weights: The request needs converted weights; cold startup may download them and matching cached weights may be reused.
  2. MLC chat request compiles when needed TVM compilation: The first-run path includes target-specific model compilation rather than assuming downloaded weights are executable.
  3. TVM compilation generates inference logic Native model library: TVM-based compilation produces the binary library consumed by the native chat runtime.
  4. Native model library is loaded into Native chat runtime: The native runtime consumes the compiled library as one of its execution inputs.
  5. Converted weights supply model parameters Native chat runtime: Converted weights supply the learned parameters, separately from the model library implementing inference.
  6. Native chat runtime encodes prompt text Prompt tokenization: The tokenizer interface converts text into token IDs before model computation.
  7. Prompt tokenization embeds prompt IDs Prompt prefill: The model interface embeds token IDs before the prompt prefill computation.
  8. Native chat runtime maintains sequence state Attention KV cache: The native model maintains historical attention key/value states for processed tokens.
  9. Attention KV cache provides sequence state Prompt prefill: Prefill is associated with a sequence in the model KV cache; the cache stores history as the request progresses.
  10. Prompt prefill returns next-token logits Token sampling: Prompt prefill produces logits used to choose the first generated token.
  11. Token sampling commits the next token Next-token decode: A subsequent decode call takes embeddings of the last committed token; generation repeats while the request remains active.
  12. Next-token decode returns new logits Token sampling: The decode interface returns model logits, which feed the next sampling decision.
  13. Token sampling decodes selected IDs Detokenized output: Selected token IDs are turned into text through the tokenizer decode interface.
  14. Prompt prefill executes compiled model on Selected local GPU: The documented first-run compiler prepares inference logic for execution on the selected local GPU.
  15. Next-token decode executes compiled model on Selected local GPU: Repeated model execution uses the compatible target-specific model library on the local GPU.

Weights are data; the model library is executable logic

A weight download is not a complete executable model. MLC documents converted weights and a model library as separate inputs. The conversion command transforms or quantizes parameters; compilation specializes inference logic for the model architecture, quantization, memory-related metadata and target platform. This page follows the documented flow rather than treating a weight file as a GPU program.

The compilation documentation says converted weights can be shared across platforms. That does not make a compiled library interchangeable between targets. Its build specification includes the platform. A CUDA or Metal build and its matching configuration must be chosen for the deployment.

Why the first run can do more work

The first-run diagram includes compilation. The cached path skips that branch and reuses a matching library, but still starts the runtime and consumes model parameters. Cache reuse here assumes the same relevant artifacts and target; it is not a guarantee that changing the architecture, quantization or configuration never triggers a rebuild.

MLC describes its compiler implementation as Python code using TVM APIs. The published serving interfaces are C++. An application may expose a Python chat API while the native runtime drives compiled model functions. Compare Transformers pipeline execution, where Python model orchestration delegates operations to PyTorch.

Prefill and decode do different jobs

Prefill consumes prompt embeddings and returns logits for the next token. Decode consumes embeddings of the last committed token and returns logits for subsequent sampling. The model interface separately exposes a KV cache maintaining the history of processed tokens. Reusing these stored attention states is different from caching compiled code.

The feedback arrow represents repeated generation while the request remains active. It does not mean the GPU computes meaningful text directly: model functions produce logits, sampling chooses token IDs, and tokenizer decoding produces text. Exact stopping rules, sampling settings and prompt formatting depend on the request and model configuration.

One request is not a batching claim

This is a logical single-request path. MLC distinguishes synchronous MLCEngine from AsyncMLCEngine and recommends the asynchronous API for concurrent request batching. Streaming describes how output is delivered to the caller; it does not, by itself, prove how many requests the engine schedules together.

For the scheduling and cache-management layers of other serving engines, compare vLLM and SGLang. Their architectures should not be copied onto MLC simply because all three generate tokens.

Every stage

Converted weights

model format

Model parameters converted or quantized into MLC-compatible format; a separate artifact from the compiled inference library.

TVM compilation

compiler

Python compiler code uses TVM APIs to produce a model library for the selected configuration and target.

In the atlas: Apache TVM

Prompt tokenization

framework component

The tokenizer wrapper encodes input text as token IDs; model token embedding turns those IDs into embeddings.

Attention KV cache

framework component

Stores historical attention keys and values for processed tokens. This is distinct from the on-disk compilation cache.

Next-token decode

framework component

Runs the last committed token through the model while retaining attention state for the sequence.

Token sampling

framework component

Sampling uses model outputs and generation configuration to choose token IDs, rather than returning a text string directly.

Detokenized output

application

The tokenizer decodes generated token IDs to text delivered by the application.

Selected local GPU

hardware

A compatible local GPU executes the target-specific compiled model. Backend and build compatibility remain required.

Every step

FromStepToWhenWhat happensSource
MLC chat requestobtains model parametersConverted weightsany startupThe request needs converted weights; cold startup may download them and matching cached weights may be reused.Source repository
MLC chat requestcompiles when neededTVM compilationFirst run / missing artifactsThe first-run path includes target-specific model compilation rather than assuming downloaded weights are executable.Source repository
TVM compilationgenerates inference logicNative model libraryFirst run / missing artifactsTVM-based compilation produces the binary library consumed by the native chat runtime.Source repository
MLC chat requestreuses matching libraryNative model libraryMatching local cacheThe cached path reuses compatible local compiled artifacts instead of traversing the compiler branch.Source repository
Native model libraryis loaded intoNative chat runtimeany startupThe native runtime consumes the compiled library as one of its execution inputs.Source repository
Converted weightssupply model parametersNative chat runtimeany startupConverted weights supply the learned parameters, separately from the model library implementing inference.Source repository
Native chat runtimeencodes prompt textPrompt tokenizationany startupThe tokenizer interface converts text into token IDs before model computation.Source repository
Prompt tokenizationembeds prompt IDsPrompt prefillany startupThe model interface embeds token IDs before the prompt prefill computation.Source repository
Native chat runtimemaintains sequence stateAttention KV cacheany startupThe native model maintains historical attention key/value states for processed tokens.Source repository
Attention KV cacheprovides sequence statePrompt prefillany startupPrefill is associated with a sequence in the model KV cache; the cache stores history as the request progresses.Source repository
Prompt prefillreturns next-token logitsToken samplingany startupPrompt prefill produces logits used to choose the first generated token.Source repository
Token samplingcommits the next tokenNext-token decodeany startupA subsequent decode call takes embeddings of the last committed token; generation repeats while the request remains active.Source repository
Next-token decodereturns new logitsToken samplingany startupThe decode interface returns model logits, which feed the next sampling decision.Source repository
Token samplingdecodes selected IDsDetokenized outputany startupSelected token IDs are turned into text through the tokenizer decode interface.Source repository
Prompt prefillexecutes compiled model onSelected local GPUany startupThe documented first-run compiler prepares inference logic for execution on the selected local GPU.Source repository
Next-token decodeexecutes compiled model onSelected local GPUany startupRepeated model execution uses the compatible target-specific model library on the local GPU.Source repository

What this does not show

One supported, text-only autoregressive model on one compatible local GPU. The startup toggle assumes an unchanged model, quantization, configuration and target: either missing compiled artifacts or a valid local cache. The diagram separates artifact dependencies from the logical generation loop; it is not a thread timeline or a claim that every kernel follows one universal backend. Distributed serving, speculative decoding, multimodal inputs, browser/mobile packaging, CPU execution and proprietary GPU-driver internals are out of scope. The GPU node represents the selected compatible target, not a claim about undocumented CUDA or Metal driver internals. The diagram is a logical dependency and token-generation view.

How this page is checked

Every stage and step above quotes the source that supports it. The quotes were checked against MLC LLM official compilation/deployment documentation and the published C++ model, tokenizer and sampler interfaces, fetched and read on 2026-11-12. on 2026-11-12. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.

See this path in the graph explorer →

What these tools are written in is in the main atlas: MLC LLM, Apache TVM. More execution paths: How Does a Rust LLM Generate Tokens with Candle, What Actually Happens When You Run JAX Code, What Runs Underneath a Keras 3 Model, How Does llama.cpp Run a Model on Your Hardware, What Actually Happens When You Run MLX on a Mac, What Actually Runs When You Send Ollama a Prompt, How Does ONNX Runtime Execute a Model Across Hardware, What Actually Happens When You Run a PyTorch Model, What Actually Happens When SGLang Serves a Request, How Does TensorFlow Run Python on a CPU or GPU, What Actually Happens When TensorRT-LLM Runs a Model, What Actually Happens When You Run a Hugging Face Pipeline, What Actually Happens When vLLM Serves a Request, or compare them all.

Explore the full graph →