How Does a Rust LLM Generate Tokens with Candle?

Candle’s Llama example runs as a Rust program. It loads model configuration and safetensors parameters, builds the Llama model, and tokenizes the prompt. The first forward pass consumes the prompt context and produces logits. With the KV cache enabled, later iterations feed one newly generated token while reusing stored attention keys and values; with caching disabled, the example feeds the accumulated token sequence again. Candle tensor storage dispatches operations to CPU, CUDA, or Metal implementations according to the selected device. A logits processor samples the next token, checks for end-of-sequence, and sends token IDs through a text output stream.
Device
KV cache

Showing cached single-token decoding after prefill on the CPU: 12 stages, 13 steps.

Rust Llama program loads parameters Safetensors weights Rust Llama program loads tokenizer.json Tokenizer Safetensors weights initializes model Candle Llama model Tokenizer encodes prompt IDs Prompt tensor Prompt tensor runs prompt forward Candle Llama model Candle Llama model updates keys and values Per-layer KV cache Per-layer KV cache reuses history Single-token decode Single-token decode runs next token Candle Llama model Candle Llama model repeats full context Full context again Full context again recomputes current sequence Candle Llama model Candle Llama model performs tensor operations Tensor computation Tensor computation uses CPU storage CPU tensor backend Tensor computation uses CUDA storage CUDA tensor backend Tensor computation uses Metal storage Metal tensor backend CPU tensor backend runs on CPU CPU CUDA tensor backend runs on NVIDIA GPU NVIDIA GPU Metal tensor backend runs on Apple GPU Apple GPU Candle Llama model returns logits for sampling Logits processor Logits processor decodes accepted tokens Text output stream Rust Llama program programming language Safetensors weights model format Tokenizer framework component Candle Llama model framework component Prompt tensor framework component Per-layer KV cache framework component Full context again framework component Single-token decode framework component Tensor computation framework component CPU tensor backend runtime CUDA tensor backend runtime Metal tensor backend runtime CPU hardware NVIDIA GPU hardware Apple GPU hardware Logits processor framework component Text output stream application

The selected path, step by step

  1. Rust Llama program loads parameters Safetensors weights: The example supplies model weight filenames, dtype and device to the safetensors-backed VarBuilder.
  2. Rust Llama program loads tokenizer.json Tokenizer: A tokenizer is constructed from the model tokenizer file before prompt encoding.
  3. Safetensors weights initializes model Candle Llama model: The loaded parameter builder and model configuration construct the Rust Llama model.
  4. Tokenizer encodes prompt IDs Prompt tensor: The tokenized context is converted to a tensor on the chosen device.
  5. Prompt tensor runs prompt forward Candle Llama model: The model runs on the input tensor, context position and current cache.
  6. Candle Llama model updates keys and values Per-layer KV cache: With caching enabled, attention records the updated key/value tensors for each block.
  7. Per-layer KV cache reuses history Single-token decode: After prefill, the loop chooses one input token when KV caching is enabled.
  8. Single-token decode runs next token Candle Llama model: The next iteration runs the model at the current context position using cached attention history.
  9. Candle Llama model performs tensor operations Tensor computation: Ordinary Llama attention includes a scaled query/key matrix multiplication.
  10. Tensor computation uses CPU storage CPU tensor backend: The storage match sends CPU-backed operands to their CPU matmul implementation.
  11. CPU tensor backend runs on CPU CPU: This branch represents the CPU device selected by the example helper.
  12. Candle Llama model returns logits for sampling Logits processor: After the forward pass and optional repetition penalty, the logits processor samples the next token.
  13. Logits processor decodes accepted tokens Text output stream: The loop sends generated token IDs to the tokenizer output stream and prints returned fragments.

The concrete program behind this page

The trace follows Candle’s published Llama example rather than combining behavior from unrelated model runners. That program retrieves configuration, tokenizer files and either a safetensors file or indexed shards, then constructs the model with a selected dtype and device. Weight loading and model computation are separate steps.

The example uses a VarBuilder backed by memory-mapped safetensors files. Its loader receives the target device and dtype explicitly. Memory mapping is a file-loading technique; the diagram’s tensor backend is the part responsible for the selected device’s computation.

The first prompt pass and the later token loop

The example tracks an index into the processed context. When caching is enabled and the iteration index is greater than zero, it selects a context size of one. Otherwise it selects the full length of the current token buffer and starts from position zero. This makes the first prompt evaluation different from subsequent cached decoding.

Inside attention, the implementation appends new key/value tensors to previously stored values and updates the per-block cache. Disabling the cache changes the work the loop feeds into the model; it does not change the model into a different tokenizer or a different set of weights. For another implementation of the prefill/decode split, see MLC LLM execution.

Device support is a build and runtime choice

The example helper honors an explicit CPU request first. Otherwise it checks CUDA availability, then Metal availability, then falls back to CPU. The source prints build-feature guidance when no GPU backend is available. The controls above show documented alternatives, not a promise that every binary contains all three backends.

The storage implementation dispatches matrix multiplication by matching CPU, CUDA or Metal storage. The selected branch calls that backend’s matmul implementation. Other tensor operators have their own implementations, and optional acceleration libraries or FlashAttention change details outside this page’s scope. Compare PyTorch’s device dispatch and MLX on Apple silicon.

Logits, sampling and readable text

A model forward pass produces logits. The example optionally applies a repetition penalty, then its logits processor samples a token and appends that ID to the running token buffer. Temperature, top-k and top-p are request settings in this example, rather than separate model implementations.

The loop checks configured end-of-sequence token IDs before sending accepted output through TokenOutputStream. A tokenizer output stream may need additional tokens before it can emit a text fragment, so one model token should not be treated as one visible word or one printed chunk. The example flushes any remaining decoded text when generation ends.

Every stage

Per-layer KV cache

framework component

Attention stores per-block key and value tensors and updates them as the sequence grows.

Full context again

framework component

Without the KV cache, each iteration feeds the full accumulated token sequence from position zero.

Single-token decode

framework component

After the first iteration, the cached path feeds one token at the current context position.

Tensor computation

framework component

The Llama attention path invokes tensor operations, including query/key matrix multiplication in the ordinary attention implementation.

In the atlas: Candle

Every step

FromStepToWhenWhat happensSource
Rust Llama programloads parametersSafetensors weightsany device · any kv cacheThe example supplies model weight filenames, dtype and device to the safetensors-backed VarBuilder.Source repository
Rust Llama programloads tokenizer.jsonTokenizerany device · any kv cacheA tokenizer is constructed from the model tokenizer file before prompt encoding.Source repository
Safetensors weightsinitializes modelCandle Llama modelany device · any kv cacheThe loaded parameter builder and model configuration construct the Rust Llama model.Source repository
Tokenizerencodes prompt IDsPrompt tensorany device · any kv cacheThe tokenized context is converted to a tensor on the chosen device.Source repository
Prompt tensorruns prompt forwardCandle Llama modelany device · any kv cacheThe model runs on the input tensor, context position and current cache.Source repository
Candle Llama modelupdates keys and valuesPer-layer KV cacheany device · KV cache enabledWith caching enabled, attention records the updated key/value tensors for each block.Source repository
Per-layer KV cachereuses historySingle-token decodeany device · KV cache enabledAfter prefill, the loop chooses one input token when KV caching is enabled.Source repository
Single-token decoderuns next tokenCandle Llama modelany device · KV cache enabledThe next iteration runs the model at the current context position using cached attention history.Source repository
Candle Llama modelrepeats full contextFull context againany device · KV cache disabledWith caching disabled, the context-selection branch uses the full accumulated token sequence.Source repository
Full context againrecomputes current sequenceCandle Llama modelany device · KV cache disabledThe model is called again with the context tensor; the uncached branch starts at position zero.Source repository
Candle Llama modelperforms tensor operationsTensor computationany device · any kv cacheOrdinary Llama attention includes a scaled query/key matrix multiplication.Source repository
Tensor computationuses CPU storageCPU tensor backendCPU · any kv cacheThe storage match sends CPU-backed operands to their CPU matmul implementation.Source repository
Tensor computationuses CUDA storageCUDA tensor backendNVIDIA GPU (CUDA) · any kv cacheThe storage match sends CUDA-backed operands to their CUDA matmul implementation.Source repository
Tensor computationuses Metal storageMetal tensor backendApple GPU (Metal) · any kv cacheThe storage match sends Metal-backed operands to their Metal matmul implementation.Source repository
CPU tensor backendruns on CPUCPUCPU · any kv cacheThis branch represents the CPU device selected by the example helper.Source repository
CUDA tensor backendruns on NVIDIA GPUNVIDIA GPUNVIDIA GPU (CUDA) · any kv cacheThis branch represents a successfully initialized CUDA device.Source repository
Metal tensor backendruns on Apple GPUApple GPUApple GPU (Metal) · any kv cacheThis branch represents a successfully initialized Metal device.Source repository
Candle Llama modelreturns logits for samplingLogits processorany device · any kv cacheAfter the forward pass and optional repetition penalty, the logits processor samples the next token.Source repository
Logits processordecodes accepted tokensText output streamany device · any kv cacheThe loop sends generated token IDs to the tokenizer output stream and prints returned fragments.Source repository

What this does not show

The published candle-examples/examples/llama floating-point, single-request inference loop with the default non-FlashAttention model path. Device options represent appropriately compiled CPU, CUDA or Metal builds; the selector does not install GPU support. CPU discussion excludes optional MKL/Accelerate builds. Quantized examples, FlashAttention, training, model-family variants, distributed serving and proprietary driver internals are out of scope. Tensor matmul is the representative backend-dispatch example, not a claim that every operator uses the same kernel. The diagram separates model control flow from representative tensor execution. Hardware nodes are sinks; arrows to sampling describe model output dependencies rather than a literal GPU scheduling timeline.

How this page is checked

Every stage and step above quotes the source that supports it. The quotes were checked against The official Candle floating-point Llama example, its Llama model implementation, storage dispatch and device-selection helper, fetched/read on 2026-11-12. on 2026-11-12. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.

See this path in the graph explorer →

What these tools are written in is in the main atlas: Rust, Safetensors, HF Tokenizers, Candle, CUDA. More execution paths: What Actually Happens When You Run JAX Code, What Runs Underneath a Keras 3 Model, How Does llama.cpp Run a Model on Your Hardware, How Does MLC LLM Turn a Compiled Model into Generated Tokens, What Actually Happens When You Run MLX on a Mac, What Actually Runs When You Send Ollama a Prompt, How Does ONNX Runtime Execute a Model Across Hardware, What Actually Happens When You Run a PyTorch Model, What Actually Happens When SGLang Serves a Request, How Does TensorFlow Run Python on a CPU or GPU, What Actually Happens When TensorRT-LLM Runs a Model, What Actually Happens When You Run a Hugging Face Pipeline, What Actually Happens When vLLM Serves a Request, or compare them all.

Explore the full graph →