{
  "version": "1.0",
  "slug": "how-candle-runs",
  "name": "Candle",
  "title": "How Candle Runs an LLM: Rust, KV Cache and GPUs | Language Lineage",
  "h1": "How Does a Rust LLM Generate Tokens with Candle?",
  "description": "Follow Candle’s Rust Llama example from safetensors and tokenization to cached decoding, sampling, and CPU, CUDA or Metal tensor computation, with quoted sources.",
  "og_subtitle": "Rust → safetensors + tokenizer → Llama → KV cache → tensor backend → tokens",
  "verified_on": "2026-11-12",
  "verified_against": "The official Candle floating-point Llama example, its Llama model implementation, storage dispatch and device-selection helper, fetched/read on 2026-11-12.",
  "scope": "The published candle-examples/examples/llama floating-point, single-request inference loop with the default non-FlashAttention model path. Device options represent appropriately compiled CPU, CUDA or Metal builds; the selector does not install GPU support. CPU discussion excludes optional MKL/Accelerate builds. Quantized examples, FlashAttention, training, model-family variants, distributed serving and proprietary driver internals are out of scope. Tensor matmul is the representative backend-dispatch example, not a claim that every operator uses the same kernel.",
  "answer": "Candle’s Llama example runs as a **Rust program**. It loads model configuration and **safetensors** parameters, builds the Llama model, and tokenizes the prompt. The first forward pass consumes the prompt context and produces logits. With the **KV cache enabled**, later iterations feed one newly generated token while reusing stored attention keys and values; with caching disabled, the example feeds the accumulated token sequence again. Candle tensor storage dispatches operations to **CPU**, **CUDA**, or **Metal** implementations according to the selected device. A logits processor samples the next token, checks for end-of-sequence, and sends token IDs through a text output stream.",
  "status_template": "Showing {cache} on {device}",
  "dimensions": [
    {
      "id": "device",
      "legend": "Device",
      "default": "cpu",
      "options": [
        {
          "value": "cpu",
          "label": "CPU",
          "phrase": "the CPU"
        },
        {
          "value": "cuda",
          "label": "NVIDIA GPU (CUDA)",
          "phrase": "an NVIDIA GPU with CUDA support"
        },
        {
          "value": "metal",
          "label": "Apple GPU (Metal)",
          "phrase": "an Apple GPU with Metal support"
        }
      ]
    },
    {
      "id": "cache",
      "legend": "KV cache",
      "default": "enabled",
      "options": [
        {
          "value": "enabled",
          "label": "KV cache enabled",
          "phrase": "cached single-token decoding after prefill"
        },
        {
          "value": "disabled",
          "label": "KV cache disabled",
          "phrase": "full-context recomputation at each iteration"
        }
      ]
    }
  ],
  "sections": [
    {
      "heading": "The concrete program behind this page",
      "paragraphs": [
        "The trace follows Candle’s published Llama example rather than combining behavior from unrelated model runners. That program retrieves configuration, tokenizer files and either a safetensors file or indexed shards, then constructs the model with a selected dtype and device. Weight loading and model computation are separate steps.",
        "The example uses a VarBuilder backed by memory-mapped safetensors files. Its loader receives the target device and dtype explicitly. Memory mapping is a file-loading technique; the diagram’s tensor backend is the part responsible for the selected device’s computation."
      ]
    },
    {
      "heading": "The first prompt pass and the later token loop",
      "paragraphs": [
        "The example tracks an index into the processed context. When caching is enabled and the iteration index is greater than zero, it selects a context size of one. Otherwise it selects the full length of the current token buffer and starts from position zero. This makes the first prompt evaluation different from subsequent cached decoding.",
        "Inside attention, the implementation appends new key/value tensors to previously stored values and updates the per-block cache. Disabling the cache changes the work the loop feeds into the model; it does not change the model into a different tokenizer or a different set of weights. For another implementation of the prefill/decode split, see [MLC LLM execution](/ai/how-mlc-llm-runs)."
      ]
    },
    {
      "heading": "Device support is a build and runtime choice",
      "paragraphs": [
        "The example helper honors an explicit CPU request first. Otherwise it checks CUDA availability, then Metal availability, then falls back to CPU. The source prints build-feature guidance when no GPU backend is available. The controls above show documented alternatives, not a promise that every binary contains all three backends.",
        "The storage implementation dispatches matrix multiplication by matching CPU, CUDA or Metal storage. The selected branch calls that backend’s matmul implementation. Other tensor operators have their own implementations, and optional acceleration libraries or FlashAttention change details outside this page’s scope. Compare [PyTorch’s device dispatch](/ai/how-pytorch-runs) and [MLX on Apple silicon](/ai/how-mlx-runs)."
      ]
    },
    {
      "heading": "Logits, sampling and readable text",
      "paragraphs": [
        "A model forward pass produces logits. The example optionally applies a repetition penalty, then its logits processor samples a token and appends that ID to the running token buffer. Temperature, top-k and top-p are request settings in this example, rather than separate model implementations.",
        "The loop checks configured end-of-sequence token IDs before sending accepted output through TokenOutputStream. A tokenizer output stream may need additional tokens before it can emit a text fragment, so one model token should not be treated as one visible word or one printed chunk. The example flushes any remaining decoded text when generation ends."
      ]
    }
  ],
  "out_of_scope_note": "The diagram separates model control flow from representative tensor execution. Hardware nodes are sinks; arrows to sampling describe model output dependencies rather than a literal GPU scheduling timeline.",
  "link_from": [
    "tool:candle",
    "lang:rust",
    "tool:safetensors",
    "tool:hf_tokenizers"
  ],
  "sources": {
    "example": {
      "url": "https://github.com/huggingface/candle/blob/main/candle-examples/examples/llama/main.rs",
      "title": "Candle floating-point Llama example: setup and token loop"
    },
    "model": {
      "url": "https://github.com/huggingface/candle/blob/main/candle-transformers/src/models/llama.rs",
      "title": "Candle Llama attention and cache implementation"
    },
    "dispatch": {
      "url": "https://github.com/huggingface/candle/blob/main/candle-core/src/storage.rs",
      "title": "Candle storage dispatch for matrix multiplication"
    },
    "device": {
      "url": "https://github.com/huggingface/candle/blob/main/candle-examples/src/lib.rs",
      "title": "Example device selection and build features"
    }
  },
  "stages": [
    {
      "id": "rust",
      "name": "Rust Llama program",
      "kind": "programming_language",
      "layer": "application",
      "lineage_id": "lang:rust",
      "pos": [
        1.5,
        0
      ],
      "summary": "The compiled Rust example coordinates model loading, a generation loop and output streaming.",
      "evidence": [
        {
          "source": "example",
          "quote": "use candle_transformers::generation::{LogitsProcessor, Sampling};"
        }
      ]
    },
    {
      "id": "weights",
      "name": "Safetensors weights",
      "kind": "model_format",
      "layer": "framework",
      "lineage_id": "tool:safetensors",
      "pos": [
        0,
        1
      ],
      "summary": "Floating-point model parameters are read through a memory-mapped safetensors VarBuilder with an explicit dtype and device.",
      "evidence": [
        {
          "source": "example",
          "quote": "VarBuilder::from_mmaped_safetensors(&filenames, dtype, &device)?"
        }
      ]
    },
    {
      "id": "tokenizer",
      "name": "Tokenizer",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": "tool:hf_tokenizers",
      "pos": [
        3,
        1
      ],
      "summary": "The example loads tokenizer.json through the Tokenizer API to encode the prompt before model execution.",
      "evidence": [
        {
          "source": "example",
          "quote": "let tokenizer = Tokenizer::from_file(tokenizer_filename).map_err(E::msg)?;"
        }
      ]
    },
    {
      "id": "model",
      "name": "Candle Llama model",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": "tool:candle",
      "pos": [
        1.5,
        2
      ],
      "summary": "The Rust Llama model is constructed from configuration and loaded parameters. Its forward pass returns logits.",
      "evidence": [
        {
          "source": "example",
          "quote": "(Llama::load(vb, &config)?, tokenizer_filename, cache, config)"
        }
      ]
    },
    {
      "id": "prompt",
      "name": "Prompt tensor",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": null,
      "pos": [
        0,
        3
      ],
      "summary": "Prompt token IDs are converted into a tensor on the selected device and given a batch dimension.",
      "evidence": [
        {
          "source": "example",
          "quote": "let input = Tensor::new(ctxt, &device)?.unsqueeze(0)?;"
        }
      ]
    },
    {
      "id": "kv",
      "name": "Per-layer KV cache",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": null,
      "pos": [
        3,
        3
      ],
      "summary": "Attention stores per-block key and value tensors and updates them as the sequence grows.",
      "evidence": [
        {
          "source": "model",
          "quote": "cache.kvs[block_idx] = Some((k.clone(), v.clone()))"
        }
      ]
    },
    {
      "id": "full_context",
      "name": "Full context again",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": null,
      "pos": [
        0,
        4
      ],
      "summary": "Without the KV cache, each iteration feeds the full accumulated token sequence from position zero.",
      "evidence": [
        {
          "source": "example",
          "quote": "let (context_size, context_index) = if cache.use_kv_cache && index > 0 {\n            (1, index_pos)\n        } else {\n            (tokens.len(), 0)\n        };"
        }
      ]
    },
    {
      "id": "incremental",
      "name": "Single-token decode",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": null,
      "pos": [
        3,
        4
      ],
      "summary": "After the first iteration, the cached path feeds one token at the current context position.",
      "evidence": [
        {
          "source": "example",
          "quote": "let (context_size, context_index) = if cache.use_kv_cache && index > 0 {\n            (1, index_pos)\n        } else {\n            (tokens.len(), 0)\n        };"
        }
      ]
    },
    {
      "id": "ops",
      "name": "Tensor computation",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": "tool:candle",
      "pos": [
        1.5,
        5
      ],
      "summary": "The Llama attention path invokes tensor operations, including query/key matrix multiplication in the ordinary attention implementation.",
      "evidence": [
        {
          "source": "model",
          "quote": "let att = (q.matmul(&k.t()?)? / (self.head_dim as f64).sqrt())?;"
        }
      ]
    },
    {
      "id": "cpu_ops",
      "name": "CPU tensor backend",
      "kind": "runtime",
      "layer": "kernel",
      "lineage_id": "tool:candle",
      "pos": [
        0,
        6
      ],
      "summary": "Matrix multiplication dispatches to the implementation for CPU-backed tensor storage.",
      "evidence": [
        {
          "source": "dispatch",
          "quote": "(Self::Cpu(lhs), Self::Cpu(rhs)) => {"
        }
      ]
    },
    {
      "id": "cuda_ops",
      "name": "CUDA tensor backend",
      "kind": "runtime",
      "layer": "kernel",
      "lineage_id": "tool:cuda",
      "pos": [
        1.5,
        6
      ],
      "summary": "Matrix multiplication dispatches to the implementation for CUDA-backed tensor storage.",
      "evidence": [
        {
          "source": "dispatch",
          "quote": "(Self::Cuda(lhs), Self::Cuda(rhs)) => {"
        }
      ]
    },
    {
      "id": "metal_ops",
      "name": "Metal tensor backend",
      "kind": "runtime",
      "layer": "kernel",
      "lineage_id": null,
      "pos": [
        3,
        6
      ],
      "summary": "Matrix multiplication dispatches to the implementation for Metal-backed tensor storage.",
      "evidence": [
        {
          "source": "dispatch",
          "quote": "(Self::Metal(lhs), Self::Metal(rhs)) => {"
        }
      ]
    },
    {
      "id": "cpu",
      "name": "CPU",
      "kind": "hardware",
      "layer": "hardware",
      "lineage_id": null,
      "pos": [
        0,
        7
      ],
      "summary": "The CPU selected explicitly or used when the example has no available GPU backend.",
      "evidence": [
        {
          "source": "device",
          "quote": "if cpu {\n        Ok(Device::Cpu)\n    }"
        }
      ]
    },
    {
      "id": "cuda",
      "name": "NVIDIA GPU",
      "kind": "hardware",
      "layer": "hardware",
      "lineage_id": null,
      "pos": [
        1.5,
        7
      ],
      "summary": "The example helper can construct CUDA device zero when CUDA support is available.",
      "evidence": [
        {
          "source": "device",
          "quote": "Ok(Device::new_cuda(0)?)"
        }
      ]
    },
    {
      "id": "metal",
      "name": "Apple GPU",
      "kind": "hardware",
      "layer": "hardware",
      "lineage_id": null,
      "pos": [
        3,
        7
      ],
      "summary": "The example helper can construct Metal device zero when Metal support is available.",
      "evidence": [
        {
          "source": "device",
          "quote": "Ok(Device::new_metal(0)?)"
        }
      ]
    },
    {
      "id": "sample",
      "name": "Logits processor",
      "kind": "framework_component",
      "layer": "framework",
      "lineage_id": null,
      "pos": [
        0.5,
        8
      ],
      "summary": "The generation loop samples a token ID from the returned logits according to its sampling configuration.",
      "evidence": [
        {
          "source": "example",
          "quote": "let next_token = logits_processor.sample(&logits)?;"
        }
      ]
    },
    {
      "id": "text",
      "name": "Text output stream",
      "kind": "application",
      "layer": "application",
      "lineage_id": null,
      "pos": [
        2.5,
        8
      ],
      "summary": "Accepted generated token IDs are decoded incrementally; the example prints fragments emitted by the output stream.",
      "evidence": [
        {
          "source": "example",
          "quote": "if let Some(t) = tokenizer.next_token(next_token)? {"
        },
        {
          "source": "example",
          "quote": "Some(model::LlamaEosToks::Single(eos_tok_id)) if next_token == eos_tok_id => {"
        }
      ]
    }
  ],
  "steps": [
    {
      "from": "rust",
      "to": "weights",
      "kind": "loads",
      "when": {},
      "label": "loads parameters",
      "explanation": "The example supplies model weight filenames, dtype and device to the safetensors-backed VarBuilder.",
      "evidence": [
        {
          "source": "example",
          "quote": "VarBuilder::from_mmaped_safetensors(&filenames, dtype, &device)?"
        }
      ]
    },
    {
      "from": "rust",
      "to": "tokenizer",
      "kind": "loads",
      "when": {},
      "label": "loads tokenizer.json",
      "explanation": "A tokenizer is constructed from the model tokenizer file before prompt encoding.",
      "evidence": [
        {
          "source": "example",
          "quote": "let tokenizer = Tokenizer::from_file(tokenizer_filename).map_err(E::msg)?;"
        }
      ]
    },
    {
      "from": "weights",
      "to": "model",
      "kind": "builds",
      "when": {},
      "label": "initializes model",
      "explanation": "The loaded parameter builder and model configuration construct the Rust Llama model.",
      "evidence": [
        {
          "source": "example",
          "quote": "(Llama::load(vb, &config)?, tokenizer_filename, cache, config)"
        }
      ]
    },
    {
      "from": "tokenizer",
      "to": "prompt",
      "kind": "calls",
      "when": {},
      "label": "encodes prompt IDs",
      "explanation": "The tokenized context is converted to a tensor on the chosen device.",
      "evidence": [
        {
          "source": "example",
          "quote": "let input = Tensor::new(ctxt, &device)?.unsqueeze(0)?;"
        }
      ]
    },
    {
      "from": "prompt",
      "to": "model",
      "kind": "calls",
      "when": {},
      "label": "runs prompt forward",
      "explanation": "The model runs on the input tensor, context position and current cache.",
      "evidence": [
        {
          "source": "example",
          "quote": "let logits = llama.forward(&input, context_index, &mut cache)?;"
        }
      ]
    },
    {
      "from": "model",
      "to": "kv",
      "kind": "builds",
      "when": {
        "cache": [
          "enabled"
        ]
      },
      "label": "updates keys and values",
      "explanation": "With caching enabled, attention records the updated key/value tensors for each block.",
      "evidence": [
        {
          "source": "model",
          "quote": "cache.kvs[block_idx] = Some((k.clone(), v.clone()))"
        }
      ]
    },
    {
      "from": "kv",
      "to": "incremental",
      "kind": "calls",
      "when": {
        "cache": [
          "enabled"
        ]
      },
      "label": "reuses history",
      "explanation": "After prefill, the loop chooses one input token when KV caching is enabled.",
      "evidence": [
        {
          "source": "example",
          "quote": "let (context_size, context_index) = if cache.use_kv_cache && index > 0 {\n            (1, index_pos)\n        } else {\n            (tokens.len(), 0)\n        };"
        }
      ]
    },
    {
      "from": "incremental",
      "to": "model",
      "kind": "calls",
      "when": {
        "cache": [
          "enabled"
        ]
      },
      "label": "runs next token",
      "explanation": "The next iteration runs the model at the current context position using cached attention history.",
      "evidence": [
        {
          "source": "example",
          "quote": "let logits = llama.forward(&input, context_index, &mut cache)?;"
        }
      ]
    },
    {
      "from": "model",
      "to": "full_context",
      "kind": "calls",
      "when": {
        "cache": [
          "disabled"
        ]
      },
      "label": "repeats full context",
      "explanation": "With caching disabled, the context-selection branch uses the full accumulated token sequence.",
      "evidence": [
        {
          "source": "example",
          "quote": "let (context_size, context_index) = if cache.use_kv_cache && index > 0 {\n            (1, index_pos)\n        } else {\n            (tokens.len(), 0)\n        };"
        }
      ]
    },
    {
      "from": "full_context",
      "to": "model",
      "kind": "calls",
      "when": {
        "cache": [
          "disabled"
        ]
      },
      "label": "recomputes current sequence",
      "explanation": "The model is called again with the context tensor; the uncached branch starts at position zero.",
      "evidence": [
        {
          "source": "example",
          "quote": "let logits = llama.forward(&input, context_index, &mut cache)?;"
        }
      ]
    },
    {
      "from": "model",
      "to": "ops",
      "kind": "calls",
      "when": {},
      "label": "performs tensor operations",
      "explanation": "Ordinary Llama attention includes a scaled query/key matrix multiplication.",
      "evidence": [
        {
          "source": "model",
          "quote": "let att = (q.matmul(&k.t()?)? / (self.head_dim as f64).sqrt())?;"
        }
      ]
    },
    {
      "from": "ops",
      "to": "cpu_ops",
      "kind": "dispatches_to",
      "when": {
        "device": [
          "cpu"
        ]
      },
      "label": "uses CPU storage",
      "explanation": "The storage match sends CPU-backed operands to their CPU matmul implementation.",
      "evidence": [
        {
          "source": "dispatch",
          "quote": "(Self::Cpu(lhs), Self::Cpu(rhs)) => {"
        }
      ]
    },
    {
      "from": "ops",
      "to": "cuda_ops",
      "kind": "dispatches_to",
      "when": {
        "device": [
          "cuda"
        ]
      },
      "label": "uses CUDA storage",
      "explanation": "The storage match sends CUDA-backed operands to their CUDA matmul implementation.",
      "evidence": [
        {
          "source": "dispatch",
          "quote": "(Self::Cuda(lhs), Self::Cuda(rhs)) => {"
        }
      ]
    },
    {
      "from": "ops",
      "to": "metal_ops",
      "kind": "dispatches_to",
      "when": {
        "device": [
          "metal"
        ]
      },
      "label": "uses Metal storage",
      "explanation": "The storage match sends Metal-backed operands to their Metal matmul implementation.",
      "evidence": [
        {
          "source": "dispatch",
          "quote": "(Self::Metal(lhs), Self::Metal(rhs)) => {"
        }
      ]
    },
    {
      "from": "cpu_ops",
      "to": "cpu",
      "kind": "executes_on",
      "when": {
        "device": [
          "cpu"
        ]
      },
      "label": "runs on CPU",
      "explanation": "This branch represents the CPU device selected by the example helper.",
      "evidence": [
        {
          "source": "device",
          "quote": "if cpu {\n        Ok(Device::Cpu)\n    }"
        }
      ]
    },
    {
      "from": "cuda_ops",
      "to": "cuda",
      "kind": "executes_on",
      "when": {
        "device": [
          "cuda"
        ]
      },
      "label": "runs on NVIDIA GPU",
      "explanation": "This branch represents a successfully initialized CUDA device.",
      "evidence": [
        {
          "source": "device",
          "quote": "Ok(Device::new_cuda(0)?)"
        }
      ]
    },
    {
      "from": "metal_ops",
      "to": "metal",
      "kind": "executes_on",
      "when": {
        "device": [
          "metal"
        ]
      },
      "label": "runs on Apple GPU",
      "explanation": "This branch represents a successfully initialized Metal device.",
      "evidence": [
        {
          "source": "device",
          "quote": "Ok(Device::new_metal(0)?)"
        }
      ]
    },
    {
      "from": "model",
      "to": "sample",
      "kind": "calls",
      "when": {},
      "label": "returns logits for sampling",
      "explanation": "After the forward pass and optional repetition penalty, the logits processor samples the next token.",
      "evidence": [
        {
          "source": "example",
          "quote": "let next_token = logits_processor.sample(&logits)?;"
        }
      ]
    },
    {
      "from": "sample",
      "to": "text",
      "kind": "calls",
      "when": {},
      "label": "decodes accepted tokens",
      "explanation": "The loop sends generated token IDs to the tokenizer output stream and prints returned fragments.",
      "evidence": [
        {
          "source": "example",
          "quote": "if let Some(t) = tokenizer.next_token(next_token)? {"
        }
      ]
    }
  ]
}
