How Does MLC LLM Turn a Compiled Model into Generated Tokens?
MLC LLM needs two different artifacts: converted model weights and a target-compatible model library containing inference logic. A first run can use Apache TVM to compile that library; later runs can reuse matching cached artifacts. The native runtime tokenizes the prompt, embeds its token IDs and performs prefill to produce next-token logits. A sampler selects a token. Subsequent decode calls process the last committed token while the KV cache retains earlier attention keys and values. That loop produces more tokens; detokenization converts their IDs back into text. GPU execution and Python compilation are distinct parts of the system.
Showing a first run with compilation for a supported single-GPU MLC model: 12 stages, 15 steps.
Your program
Runtime
Framework core
Model file
Compiler
Hardware
The selected path, step by step
MLC chat request obtains model parameters Converted weights: The request needs converted weights; cold startup may download them and matching cached weights may be reused.
MLC chat request compiles when needed TVM compilation: The first-run path includes target-specific model compilation rather than assuming downloaded weights are executable.
TVM compilation generates inference logic Native model library: TVM-based compilation produces the binary library consumed by the native chat runtime.
Native model library is loaded into Native chat runtime: The native runtime consumes the compiled library as one of its execution inputs.
Converted weights supply model parameters Native chat runtime: Converted weights supply the learned parameters, separately from the model library implementing inference.
Native chat runtime encodes prompt text Prompt tokenization: The tokenizer interface converts text into token IDs before model computation.
Prompt tokenization embeds prompt IDs Prompt prefill: The model interface embeds token IDs before the prompt prefill computation.
Native chat runtime maintains sequence state Attention KV cache: The native model maintains historical attention key/value states for processed tokens.
Attention KV cache provides sequence state Prompt prefill: Prefill is associated with a sequence in the model KV cache; the cache stores history as the request progresses.
Prompt prefill returns next-token logits Token sampling: Prompt prefill produces logits used to choose the first generated token.
Token sampling commits the next token Next-token decode: A subsequent decode call takes embeddings of the last committed token; generation repeats while the request remains active.
Next-token decode returns new logits Token sampling: The decode interface returns model logits, which feed the next sampling decision.
Token sampling decodes selected IDs Detokenized output: Selected token IDs are turned into text through the tokenizer decode interface.
Prompt prefill executes compiled model on Selected local GPU: The documented first-run compiler prepares inference logic for execution on the selected local GPU.
Next-token decode executes compiled model on Selected local GPU: Repeated model execution uses the compatible target-specific model library on the local GPU.
Weights are data; the model library is executable logic
A weight download is not a complete executable model. MLC documents converted weights and a model library as separate inputs. The conversion command transforms or quantizes parameters; compilation specializes inference logic for the model architecture, quantization, memory-related metadata and target platform. This page follows the documented flow rather than treating a weight file as a GPU program.
The compilation documentation says converted weights can be shared across platforms. That does not make a compiled library interchangeable between targets. Its build specification includes the platform. A CUDA or Metal build and its matching configuration must be chosen for the deployment.
Why the first run can do more work
The first-run diagram includes compilation. The cached path skips that branch and reuses a matching library, but still starts the runtime and consumes model parameters. Cache reuse here assumes the same relevant artifacts and target; it is not a guarantee that changing the architecture, quantization or configuration never triggers a rebuild.
MLC describes its compiler implementation as Python code using TVM APIs. The published serving interfaces are C++. An application may expose a Python chat API while the native runtime drives compiled model functions. Compare Transformers pipeline execution, where Python model orchestration delegates operations to PyTorch.
Prefill and decode do different jobs
Prefill consumes prompt embeddings and returns logits for the next token. Decode consumes embeddings of the last committed token and returns logits for subsequent sampling. The model interface separately exposes a KV cache maintaining the history of processed tokens. Reusing these stored attention states is different from caching compiled code.
The feedback arrow represents repeated generation while the request remains active. It does not mean the GPU computes meaningful text directly: model functions produce logits, sampling chooses token IDs, and tokenizer decoding produces text. Exact stopping rules, sampling settings and prompt formatting depend on the request and model configuration.
One request is not a batching claim
This is a logical single-request path. MLC distinguishes synchronous MLCEngine from AsyncMLCEngine and recommends the asynchronous API for concurrent request batching. Streaming describes how output is delivered to the caller; it does not, by itself, prove how many requests the engine schedules together.
For the scheduling and cache-management layers of other serving engines, compare vLLM and SGLang. Their architectures should not be copied onto MLC simply because all three generate tokens.
Every stage
MLC chat request
application
A text-generation request through the documented MLC chat API or CLI. The page follows one request, not a batch scheduler.
One supported, text-only autoregressive model on one compatible local GPU. The startup toggle assumes an unchanged model, quantization, configuration and target: either missing compiled artifacts or a valid local cache. The diagram separates artifact dependencies from the logical generation loop; it is not a thread timeline or a claim that every kernel follows one universal backend. Distributed serving, speculative decoding, multimodal inputs, browser/mobile packaging, CPU execution and proprietary GPU-driver internals are out of scope. The GPU node represents the selected compatible target, not a claim about undocumented CUDA or Metal driver internals. The diagram is a logical dependency and token-generation view.
How this page is checked
Every stage and step above quotes the source that supports it. The quotes were checked against MLC LLM official compilation/deployment documentation and the published C++ model, tokenizer and sampler interfaces, fetched and read on 2026-11-12. on 2026-11-12. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.