Showing the CUDA execution provider: 10 stages, 9 steps.
ONNX Runtime splits graph transformations into levels that run before and after hardware partitioning. Basic graph optimizations—constant folding, dead-node elimination, and simple operator fusions like Conv+Add—are semantics-preserving rewrites that run before partitioning, so every execution provider benefits from them.
Extended graph optimizations (such as multi-node Attention, LayerNorm, and Bias+GELU fusions) and NCHWc layout transformations run after GetCapability() partitioning, because they emit fused operator nodes implemented specifically by the CPU, CUDA, or ROCm execution providers.
Unlike a single-backend engine, an ONNX Runtime session takes a priority-ordered list of execution providers—for example, TensorrtExecutionProvider, then CUDAExecutionProvider, then CPUExecutionProvider. ONNX Runtime queries each provider's GetCapability() interface in order to claim subgraphs or individual nodes.
Any operator that a higher-priority accelerator provider cannot run is automatically assigned to the next provider in the list, with the built-in CPU execution provider acting as the final fallback.
application
Your Python, C++, C#, C, Java, or Node.js code that configures an ONNX Runtime session with a priority-ordered list of execution providers.
In the atlas: Python
The APIs to set EP options are available across Python, C/C++/C#, Java and node.js.ONNX Runtime Execution Providers (ONNX Runtime documentation) Project docs
The list of providers is ordered by Priority.ONNX Runtime Execution Providers (ONNX Runtime documentation) Project docs
model format
A serialized ONNX computation graph containing standard operators, tensor types, and constant weight initializers exported from PyTorch, TensorFlow, or scikit-learn.
ONNX provides an open source format for AI models, both deep learning and traditional ML. It defines an extensible computation graph model, as well as definitions of built-in operators and standard data types.ONNX README (onnx/onnx) Source repository
supporting models from deep learning frameworks such as PyTorch and TensorFlow/Keras as well as classical machine learning libraries such as scikit-learn, LightGBM, XGBoost, etc.ONNX Runtime README (microsoft/onnxruntime) Source repository
runtime
ONNX Runtime's C++ session engine. When initialized, it parses the model into an in-memory graph, runs graph optimizations, and coordinates execution providers.
In the atlas: C++
ONNX Runtime is compatible with different hardware, drivers, and operating systems, and provides optimal performance by leveraging hardware accelerators where applicable alongside graph optimizations and transforms.ONNX Runtime README (microsoft/onnxruntime) Source repository
In online mode, when initializing an inference session, we also apply all enabled graph optimizations before performing model inference.Graph Optimizations in ONNX Runtime (ONNX Runtime documentation) Project docs
compiler
Level-1 semantics-preserving graph rewrites—constant folding, redundant node elimination, and Conv/Add/BatchNorm fusions—applied before partitioning.
These are semantics-preserving graph rewrites which remove redundant nodes and redundant computation. They run before graph partitioning and thus apply to all the execution providers.Graph Optimizations in ONNX Runtime (ONNX Runtime documentation) Project docs
Constant Folding: Statically computes parts of the graph that rely only on constant initializers.Graph Optimizations in ONNX Runtime (ONNX Runtime documentation) Project docs
framework component
Queries registered execution providers via GetCapability() in priority order to allocate supported subgraphs or nodes to each provider.
In the atlas: C++
ONNX Runtime works with the execution provider(s) using the GetCapability() interface to allocate specific nodes or sub-graphs for execution by the EP library in supported hardware.ONNX Runtime Execution Providers (ONNX Runtime documentation) Project docs
compiler
Post-partitioning complex operator fusions (Attention, LayerNormalization, SkipLayerNorm, BiasGelu on CPU/CUDA) and NCHWc data layout transforms on CPU.
These optimizations include complex node fusions. They are run after graph partitioning and are only applied to the nodes assigned to the CPU or CUDA or ROCm execution provider.Graph Optimizations in ONNX Runtime (ONNX Runtime documentation) Project docs
NCHWc Optimizer: Optimizes the graph by using NCHWc layout instead of NCHW layout.Graph Optimizations in ONNX Runtime (ONNX Runtime documentation) Project docs
compiler
Compiles supported ONNX subgraphs into optimized NVIDIA TensorRT engines for the target GPU.
The TensorRT execution provider in the ONNX Runtime makes use of NVIDIA’s TensorRT Deep Learning inferencing engine to accelerate ONNX model in their family of GPUs.TensorRT Execution Provider (ONNX Runtime documentation) Project docs
kernel
Runs assigned operators on the NVIDIA GPU using precompiled CUDA kernels and NVIDIA libraries including cuDNN, cuBLAS, and cuBLASLt.
In the atlas: CUDA
The CUDA Execution Provider enables hardware accelerated computation on Nvidia CUDA-enabled GPUs.CUDA Execution Provider (ONNX Runtime documentation) Project docs
libcudart 11.4.43 libcufft 10.5.2.100 libcurand 10.2.5.120 libcublasLt 11.6.5.2 libcublas 11.6.5.2 libcudnn 8.2.4CUDA Execution Provider (ONNX Runtime documentation) Project docs
kernel
ONNX Runtime's built-in C++ CPU execution provider. Runs the full graph on CPU or any operator nodes not claimed by a higher-priority GPU provider.
In the atlas: C++
The ONNX Runtime package can be built with any combination of the EPs along with the default CPU execution provider.ONNX Runtime Execution Providers (ONNX Runtime documentation) Project docs
For example ['CUDAExecutionProvider', 'CPUExecutionProvider'] means execute a node using CUDAExecutionProvider if capable, otherwise execute using CPUExecutionProvider.ONNX Runtime Execution Providers (ONNX Runtime documentation) Project docs
hardware
The host CPU executing the CPU execution provider's C++ kernels.
This architecture abstracts out the details of the hardware specific libraries that are essential to optimize the execution of deep neural networks across hardware platforms like CPU, GPU, FPGA or specialized NPUs.ONNX Runtime Execution Providers (ONNX Runtime documentation) Project docs
hardware
The NVIDIA GPU executing compiled TensorRT engines and CUDA / cuDNN / cuBLAS kernels.
The CUDA Execution Provider enables hardware accelerated computation on Nvidia CUDA-enabled GPUs.CUDA Execution Provider (ONNX Runtime documentation) Project docs
| From | Step | To | When | What happens | Source |
|---|---|---|---|---|---|
| Your application | passes model path to | ONNX model (.onnx) | any execution provider | Your application passes the .onnx model file (or bytes) and the priority-ordered execution provider list when creating an InferenceSession. | Project docs |
| ONNX model (.onnx) | loaded into | InferenceSession | any execution provider | InferenceSession parses the ONNX protobuf graph, operator nodes, and initializers into ONNX Runtime's in-memory Graph representation. | Project docs |
| InferenceSession | runs level-1 passes in | Basic graph passes | any execution provider | Before partitioning the graph across providers, ONNX Runtime applies Basic graph optimizations such as constant folding, dead-node elimination, and Conv+Add fusion. | Project docs |
| Basic graph passes | handed to | GetCapability split | any execution provider | The simplified graph is passed to ONNX Runtime's graph partitioner to assign nodes or subgraphs to registered execution providers. | Project docs |
| GetCapability split | fuses CPU/CUDA nodes in | Extended & layout opt | CPU (Default) or CUDA (NVIDIA) | For nodes assigned to the CPU or CUDA execution providers, ONNX Runtime runs Extended graph optimizations (complex node fusions) and CPU layout optimizations. | Project docs |
| Extended & layout opt | dispatches fused graph to | CPU EP kernels | CPU (Default) | When using the default CPU execution provider, all optimized and layout-transformed nodes are executed by the CPU EP's C++ kernels. | Project docs |
| Extended & layout opt | dispatches GPU nodes to | CUDA EP (cuDNN/CUDA) | CUDA (NVIDIA) | Nodes claimed by CUDAExecutionProvider (including fused Attention, LayerNorm, and BiasGelu nodes) run through the CUDA execution provider. | Project docs |
| GetCapability split | allocates subgraphs to | TensorRT EP | TensorRT (NVIDIA) | With TensorrtExecutionProvider registered first, GetCapability() assigns supported ONNX subgraphs to TensorRT to be compiled into TensorRT engines. | Project docs |
| TensorRT EP | runs TensorRT engine on | NVIDIA GPU | TensorRT (NVIDIA) | The compiled TensorRT engine executes its fused layers directly on the NVIDIA GPU. | Project docs |
| TensorRT EP | unsupported GPU ops fall back to | CUDA EP (cuDNN/CUDA) | TensorRT (NVIDIA) | Operators that TensorRT does not support are assigned to the second-priority CUDAExecutionProvider rather than dropping all the way to the CPU. | Project docs |
| GetCapability split | unclaimed ops fall back to | CPU EP kernels | CUDA (NVIDIA) or TensorRT (NVIDIA) | Any operator node not supported by the GPU execution providers is assigned to the built-in CPUExecutionProvider. | Project docs |
| CUDA EP (cuDNN/CUDA) | launches CUDA/cuDNN kernels on | NVIDIA GPU | CUDA (NVIDIA) or TensorRT (NVIDIA) | The CUDA execution provider launches CUDA, cuDNN, and cuBLAS kernels on the NVIDIA GPU. | Project docs |
| CPU EP kernels | runs on | CPU | any execution provider | The CPU execution provider runs its C++ kernels on the host CPU. | Project docs |
Documented inference paths for an ONNX (.onnx) model in ONNX Runtime using three execution providers: the default CPU provider, the NVIDIA CUDA provider, and the NVIDIA TensorRT provider. Other execution providers (CoreML, DirectML, OpenVINO, QNN, ROCm, WebGPU) and ONNX Runtime Training are out of scope. ONNX Runtime uses the same GetCapability() partitioning architecture for more than a dozen other execution providers, including CoreML, DirectML, OpenVINO, and WebGPU.
Every stage and step above quotes the source that supports it. The quotes were checked against the ONNX specification README and the ONNX Runtime documentation (Execution Providers, Graph Optimizations, CUDA Execution Provider, TensorRT Execution Provider) on 2026-11-11. The data is a versioned JSON file under the same CC BY 4.0 license as the dataset. Found something wrong or out of date? Open an issue with the better source.
What these tools are written in is in the main atlas: Python, C++, CUDA. More execution paths: What Actually Happens When You Run JAX Code, How Does llama.cpp Run a Model on Your Hardware, What Actually Happens When You Run MLX on a Mac, What Actually Happens When You Run a PyTorch Model, What Actually Happens When vLLM Serves a Request, or compare them all.
Explore the full graph →