Candle’s Llama example runs as a Rust program. It loads model configuration and safetensors parameters, builds the Llama model, and tokenizes the prompt. The first forward pass consumes the prompt context and produces logits. With the KV cache enabled, later iterations feed one newly generated token while reusing stored attention keys and values; with caching disabled, the example feeds the accumulated token sequence again. Candle tensor storage dispatches operations to CPU, CUDA, or Metal implementations according to the selected device. A logits processor samples the next token, checks for end-of-sequence, and sends token IDs through a text output stream.
Showing cached single-token decoding after prefill on the CPU: 12 stages, 13 steps.
Your program
Runtime
Framework core
Model file
Hardware
The selected path, step by step
Rust Llama program loads parameters Safetensors weights: The example supplies model weight filenames, dtype and device to the safetensors-backed VarBuilder.
Rust Llama program loads tokenizer.json Tokenizer: A tokenizer is constructed from the model tokenizer file before prompt encoding.
Safetensors weights initializes model Candle Llama model: The loaded parameter builder and model configuration construct the Rust Llama model.
Tokenizer encodes prompt IDs Prompt tensor: The tokenized context is converted to a tensor on the chosen device.
Prompt tensor runs prompt forward Candle Llama model: The model runs on the input tensor, context position and current cache.
Candle Llama model updates keys and values Per-layer KV cache: With caching enabled, attention records the updated key/value tensors for each block.
Per-layer KV cache reuses history Single-token decode: After prefill, the loop chooses one input token when KV caching is enabled.
Single-token decode runs next token Candle Llama model: The next iteration runs the model at the current context position using cached attention history.
Candle Llama model performs tensor operations Tensor computation: Ordinary Llama attention includes a scaled query/key matrix multiplication.
Tensor computation uses CPU storage CPU tensor backend: The storage match sends CPU-backed operands to their CPU matmul implementation.
CPU tensor backend runs on CPU CPU: This branch represents the CPU device selected by the example helper.
Candle Llama model returns logits for sampling Logits processor: After the forward pass and optional repetition penalty, the logits processor samples the next token.
Logits processor decodes accepted tokens Text output stream: The loop sends generated token IDs to the tokenizer output stream and prints returned fragments.
The concrete program behind this page
The trace follows Candle’s published Llama example rather than combining behavior from unrelated model runners. That program retrieves configuration, tokenizer files and either a safetensors file or indexed shards, then constructs the model with a selected dtype and device. Weight loading and model computation are separate steps.
The example uses a VarBuilder backed by memory-mapped safetensors files. Its loader receives the target device and dtype explicitly. Memory mapping is a file-loading technique; the diagram’s tensor backend is the part responsible for the selected device’s computation.
The first prompt pass and the later token loop
The example tracks an index into the processed context. When caching is enabled and the iteration index is greater than zero, it selects a context size of one. Otherwise it selects the full length of the current token buffer and starts from position zero. This makes the first prompt evaluation different from subsequent cached decoding.
Inside attention, the implementation appends new key/value tensors to previously stored values and updates the per-block cache. Disabling the cache changes the work the loop feeds into the model; it does not change the model into a different tokenizer or a different set of weights. For another implementation of the prefill/decode split, see MLC LLM execution.
Device support is a build and runtime choice
The example helper honors an explicit CPU request first. Otherwise it checks CUDA availability, then Metal availability, then falls back to CPU. The source prints build-feature guidance when no GPU backend is available. The controls above show documented alternatives, not a promise that every binary contains all three backends.
The storage implementation dispatches matrix multiplication by matching CPU, CUDA or Metal storage. The selected branch calls that backend’s matmul implementation. Other tensor operators have their own implementations, and optional acceleration libraries or FlashAttention change details outside this page’s scope. Compare PyTorch’s device dispatch and MLX on Apple silicon.
Logits, sampling and readable text
A model forward pass produces logits. The example optionally applies a repetition penalty, then its logits processor samples a token and appends that ID to the running token buffer. Temperature, top-k and top-p are request settings in this example, rather than separate model implementations.
The loop checks configured end-of-sequence token IDs before sending accepted output through TokenOutputStream. A tokenizer output stream may need additional tokens before it can emit a text fragment, so one model token should not be treated as one visible word or one printed chunk. The example flushes any remaining decoded text when generation ends.
Every stage
Rust Llama program
programming language
The compiled Rust example coordinates model loading, a generation loop and output streaming.
The published candle-examples/examples/llama floating-point, single-request inference loop with the default non-FlashAttention model path. Device options represent appropriately compiled CPU, CUDA or Metal builds; the selector does not install GPU support. CPU discussion excludes optional MKL/Accelerate builds. Quantized examples, FlashAttention, training, model-family variants, distributed serving and proprietary driver internals are out of scope. Tensor matmul is the representative backend-dispatch example, not a claim that every operator uses the same kernel. The diagram separates model control flow from representative tensor execution. Hardware nodes are sinks; arrows to sampling describe model output dependencies rather than a literal GPU scheduling timeline.
How this page is checked
Every stage and step above quotes the source that supports it. The quotes were checked against The official Candle floating-point Llama example, its Llama model implementation, storage dispatch and device-selection helper, fetched/read on 2026-11-12. on 2026-11-12. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.