How Does llama.cpp Run a Model on Your Hardware?

llama.cpp is plain C/C++: your program calls the llama.cpp library, which loads a GGUF model file and builds a ggml compute graph for each step of inference. ggml then runs that graph on a backend. On the CPU it uses SIMD kernels written in C/C++; on an NVIDIA GPU it runs CUDA kernels that nvcc compiled when llama.cpp was built; on Apple silicon it runs kernels written in Metal Shading Language. With a GPU backend, layers that are not offloaded to the GPU still run on the CPU.

Showing the Metal (Apple silicon) backend: 9 stages, 8 steps.

Your program
Framework core
Model file
Kernel
Compiler
Hardware

The selected path, step by step

Your program calls llama.cpp library: The program calls the llama.cpp library, which is plain C/C++ with no other dependencies.
llama.cpp library loads GGUF model file: The library reads the model's weights and metadata from a GGUF file.
llama.cpp library builds ggml compute graph: For each step of generation, llama.cpp describes the computation as a ggml graph.
ggml compute graph runs layers on ggml CPU backend: With the CPU backend the whole graph runs here. With a GPU backend, the layers not offloaded to the GPU still run here; --n-gpu-layers controls the split.
ggml compute graph offloads layers to ggml Metal backend: On macOS builds Metal is on by default, and the offloaded layers are computed by the Metal backend. --n-gpu-layers 0 turns GPU inference off.
ggml CPU backend runs on CPU: The C/C++ kernels run on the CPU's cores using its SIMD instructions.
ggml Metal backend kernels written in Metal kernels (MSL): The Metal backend's operations are llama.cpp's own kernels in Metal Shading Language.
Metal kernels (MSL) run on Apple silicon GPU: Through Apple's Metal framework the kernels run on the Apple silicon GPU.

Which language does the work

There is no Python on this path. The library, the model loader and ggml are C and C++, and the language of the innermost loops depends on the backend: C/C++ with SIMD instructions on the CPU, CUDA C++ on NVIDIA GPUs, and Metal Shading Language on Apple GPUs.

The GPU kernels are written for each backend and kept in the llama.cpp repository. CUDA kernels are compiled by NVIDIA's nvcc when llama.cpp itself is built, by default for the GPU in the build machine. That is a different arrangement from torch.compile, where TorchInductor generates new kernels while your program runs.

CPU and GPU together

With the CUDA or Metal backend, the GPU only runs the layers offloaded to it, which the --n-gpu-layers (-ngl) option controls. The rest of the model runs on the CPU backend, which is how llama.cpp runs models larger than the GPU's memory.

Every stage

Your program

application

A program that runs a model: llama.cpp's own command-line tool or OpenAI-compatible server, or your own code linked against the library.

In the atlas: llama.cpp

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware
llama.cpp README Source repository
Launch OpenAI-compatible API server
llama.cpp README Source repository

llama.cpp library

framework component

The C/C++ inference library: it loads the model and builds the computation for each step of generation on top of ggml.

In the atlas: llama.cpp

Plain C/C++ implementation without any dependencies
llama.cpp README Source repository
LLM inference with llama.cpp, which builds upon ggml.
Introduction to ggml (Hugging Face blog, by ggml maintainers) Project docs

GGUF model file

model format

The model's weights and metadata in one binary file. GGUF is a file format, not a programming language.

GGUF is a file format for storing models for inference with GGML and executors based on GGML.
GGUF specification (ggml) Source repository

ggml compute graph

framework component

ggml, the C/C++ tensor library under llama.cpp, records the operations of each step as a graph, then hands it to a backend to compute.

In the atlas: ggml

ggml is a machine learning (ML) library written in C and C++ with a focus on Transformer inference.
Introduction to ggml (Hugging Face blog, by ggml maintainers) Project docs
ggml_cgraph : Represents a computational graph. Think of it as the "order of computation" that will be transferred to the backend.
Introduction to ggml (Hugging Face blog, by ggml maintainers) Project docs

ggml CPU backend

kernel

C/C++ kernels using the CPU's SIMD instructions: AVX on x86, NEON on ARM, including Apple silicon.

In the atlas: ggml

AVX, AVX2, AVX512 and AMX support for x86 architectures
llama.cpp README Source repository
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
llama.cpp README Source repository

ggml CUDA backend

framework component

The backend for NVIDIA GPUs, enabled at build time with -DGGML_CUDA=ON. It runs llama.cpp's own CUDA kernels.

In the atlas: ggml

Custom CUDA kernels for running LLMs on NVIDIA GPUs
llama.cpp README Source repository
This provides GPU acceleration using an NVIDIA GPU.
llama.cpp build guide Source repository
cmake -B build -DGGML_CUDA=ON
llama.cpp build guide Source repository

ggml Metal backend

framework component

The backend for Apple GPUs, enabled by default on macOS. It runs llama.cpp's own Metal kernels through Apple's Metal framework.

On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU.
llama.cpp build guide Source repository
Apple silicon is a first-class citizen - optimized via ARM NEON, Accelerate and Metal frameworks
llama.cpp README Source repository

CUDA kernels (.cu)

kernel language

GPU functions written in CUDA C++ and marked __global__, in .cu files under ggml/src/ggml-cuda (for example argmax.cu).

In the atlas: CUDA

static __global__ void argmax_f32(const float * __restrict__ x, int32_t * __restrict__ dst, const int64_t ncols)
llama.cpp CUDA kernel source (ggml-cuda/argmax.cu) Source repository

Metal kernels (MSL)

kernel language

GPU functions written in Metal Shading Language and marked kernel, in .metal files under ggml/src/ggml-metal/kernels.

kernel void kernel_argsort_f32_i32(
llama.cpp Metal kernel source (ggml-metal/kernels/argsort.metal) Source repository

nvcc (CUDA toolkit)

compiler

NVIDIA's CUDA compiler. It compiles the CUDA kernels when llama.cpp is built, by default for the GPU in the build machine.

In the atlas: CUDA

If nvcc cannot detect your gpu, you may get compile warnings such as
llama.cpp build guide Source repository
By default llama.cpp will be built for the hardware that is connected to the system at that time.
llama.cpp build guide Source repository

CPU

hardware

Runs the whole model with the CPU backend, or the layers not offloaded to a GPU.

CPU+GPU hybrid inference to partially accelerate models larger than the total VRAM capacity
llama.cpp README Source repository

NVIDIA GPU

hardware

Runs the compiled CUDA kernels for the offloaded layers.

This provides GPU acceleration using an NVIDIA GPU.
llama.cpp build guide Source repository

Apple silicon GPU

hardware

Runs the Metal kernels for the offloaded layers.

On MacOS, Metal is enabled by default. Using Metal makes the computation run on the GPU.
llama.cpp build guide Source repository

Every step

From	Step	To	When	What happens	Source
Your program	calls	llama.cpp library	any backend	The program calls the llama.cpp library, which is plain C/C++ with no other dependencies.	Source repository
llama.cpp library	loads	GGUF model file	any backend	The library reads the model's weights and metadata from a GGUF file.	Source repository
llama.cpp library	builds	ggml compute graph	any backend	For each step of generation, llama.cpp describes the computation as a ggml graph.	Project docs Project docs
ggml compute graph	runs layers on	ggml CPU backend	any backend	With the CPU backend the whole graph runs here. With a GPU backend, the layers not offloaded to the GPU still run here; --n-gpu-layers controls the split.	Source repository Project docs
ggml compute graph	offloads layers to	ggml CUDA backend	CUDA (NVIDIA)	In a CUDA build, the layers offloaded to the GPU are computed by the CUDA backend.	Project docs Source repository
ggml compute graph	offloads layers to	ggml Metal backend	Metal (Apple)	On macOS builds Metal is on by default, and the offloaded layers are computed by the Metal backend. --n-gpu-layers 0 turns GPU inference off.	Project docs Source repository
ggml CPU backend	runs on	CPU	any backend	The C/C++ kernels run on the CPU's cores using its SIMD instructions.	Source repository
ggml CUDA backend	kernels written in	CUDA kernels (.cu)	CUDA (NVIDIA)	The CUDA backend's operations are llama.cpp's own CUDA C++ kernels.	Source repository Source repository
CUDA kernels (.cu)	compiled at build time by	nvcc (CUDA toolkit)	CUDA (NVIDIA)	The kernels are compiled by NVIDIA's nvcc when llama.cpp is built, by default for the GPU present at build time.	Source repository Source repository
nvcc (CUDA toolkit)	compiles for	NVIDIA GPU	CUDA (NVIDIA)	The compiled kernels run on the NVIDIA GPU. A build made for all CUDA GPUs may need some just-in-time compilation on the target GPU.	Source repository
ggml Metal backend	kernels written in	Metal kernels (MSL)	Metal (Apple)	The Metal backend's operations are llama.cpp's own kernels in Metal Shading Language.	Source repository Source repository
Metal kernels (MSL)	run on	Apple silicon GPU	Metal (Apple)	Through Apple's Metal framework the kernels run on the Apple silicon GPU.	Source repository

What this does not show

Documented paths for running a GGUF model with llama.cpp on three of its backends: the CPU, CUDA on an NVIDIA GPU, and Metal on Apple silicon. Other backends (Vulkan, HIP, SYCL, OpenCL and more), multi-GPU and RPC setups, and how a model is converted to GGUF are out of scope. The same ggml graph can run on any backend the build enables; this page shows three of them.

How this page is checked

Every stage and step above quotes the source that supports it. The quotes were checked against the llama.cpp README and build guide, the GGUF specification, Hugging Face's Introduction to ggml, and llama.cpp's CUDA and Metal kernel sources (master branch) on 2026-11-11. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.

What these tools are written in is in the main atlas: llama.cpp, ggml, CUDA. More execution paths: What Actually Happens When You Run a PyTorch Model.

Explore the full graph →