ONNX Model Interchange Cheat Sheet
Export, inspect, and run models across frameworks using the Open Neural Network Exchange format and the ONNX Runtime.
2 PagesIntermediateMar 3, 2026
Export a PyTorch Model to ONNX
Trace a model and write it to the ONNX format with dynamic batch axes.
python
import torchmodel.eval()dummy_input = torch.randn(1, 3, 224, 224)torch.onnx.export( model, dummy_input, "model.onnx", input_names=["input"], output_names=["output"], dynamic_axes={"input": {0: "batch_size"}, "output": {0: "batch_size"}}, opset_version=18,)
Export a scikit-learn Model
Convert a trained sklearn pipeline into ONNX using skl2onnx.
python
from skl2onnx import to_onnxfrom skl2onnx.common.data_types import FloatTensorTypeinitial_type = [("input", FloatTensorType([None, X_train.shape[1]]))]onnx_model = to_onnx(clf, initial_types=initial_type)with open("clf.onnx", "wb") as f: f.write(onnx_model.SerializeToString())
Run Inference with ONNX Runtime
Load an .onnx file and run a prediction with the ONNX Runtime session API.
python
import onnxruntime as ortimport numpy as npsess = ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider", "CPUExecutionProvider"])input_name = sess.get_inputs()[0].nameoutput_name = sess.get_outputs()[0].namex = np.random.randn(1, 3, 224, 224).astype(np.float32)result = sess.run([output_name], {input_name: x})print(result[0].shape)
Inspect and Optimize a Graph
Validate the model, print its graph, and apply graph-level optimizations before deployment.
bash
# validate the model structurepython -c "import onnx; m = onnx.load('model.onnx'); onnx.checker.check_model(m); print('valid')"# human-readable graph dumppython -c "import onnx; print(onnx.helper.printable_graph(onnx.load('model.onnx').graph))"# quantize to int8 for faster CPU inferencepython -m onnxruntime.quantization.preprocess --input model.onnx --output model-pre.onnxpython -c "from onnxruntime.quantization import quantize_dynamic, QuantType; \quantize_dynamic('model-pre.onnx', 'model-int8.onnx', weight_type=QuantType.QInt8)"
Execution Providers
Hardware backends ONNX Runtime can target via the providers list, tried in order.
- CPUExecutionProvider- default fallback, runs on any machine
- CUDAExecutionProvider- NVIDIA GPU inference via CUDA/cuDNN
- TensorrtExecutionProvider- NVIDIA TensorRT for lower-latency GPU inference
- CoreMLExecutionProvider- Apple Silicon/Neural Engine acceleration
- OpenVINOExecutionProvider- Intel CPU/iGPU/VPU acceleration
- DmlExecutionProvider- DirectML backend for Windows GPUs
Zero-Copy GPU Inference with IOBinding
Bind input/output tensors directly to GPU memory to avoid host<->device copies on every call.
python
import onnxruntime as ortimport numpy as npimport torchsess = ort.InferenceSession("model.onnx", providers=["CUDAExecutionProvider"])io_binding = sess.io_binding()x = torch.randn(8, 3, 224, 224, device="cuda", dtype=torch.float32)io_binding.bind_input( name="input", device_type="cuda", device_id=0, element_type=np.float32, shape=tuple(x.shape), buffer_ptr=x.data_ptr(),)io_binding.bind_output("output", device_type="cuda")sess.run_with_iobinding(io_binding)outputs = io_binding.copy_outputs_to_cpu()
Tune Session Options for Throughput
Configure graph optimization level, execution mode, and threading before opening a session.
python
import onnxruntime as ortopts = ort.SessionOptions()opts.graph_optimization_level = ort.GraphOptimizationLevel.ORT_ENABLE_ALLopts.execution_mode = ort.ExecutionMode.ORT_PARALLELopts.intra_op_num_threads = 4opts.inter_op_num_threads = 2opts.optimized_model_filepath = "model.optimized.onnx"sess = ort.InferenceSession("model.onnx", sess_options=opts, providers=["CPUExecutionProvider"])
Save/Load Models Above 2GB
Protobuf caps a single .onnx file at 2GB, so large weights must be split into an external data file.
python
import onnxmodel = onnx.load("model.onnx", load_external_data=True)onnx.save_model( model, "model-external.onnx", save_as_external_data=True, all_tensors_to_one_file=True, location="model-external.data", size_threshold=1024, convert_attribute=False,)
Static Quantization with a Calibration Reader
Static QDQ quantization uses real calibration data for tighter accuracy than dynamic quantization.
python
from onnxruntime.quantization import ( CalibrationDataReader, quantize_static, QuantFormat, QuantType,)class Calib(CalibrationDataReader): def __init__(self, samples): self._iter = iter({"input": s} for s in samples) def get_next(self): return next(self._iter, None)quantize_static( "model-pre.onnx", "model-int8-static.onnx", calibration_data_reader=Calib(calib_samples), quant_format=QuantFormat.QDQ, activation_type=QuantType.QInt8, weight_type=QuantType.QInt8,)
Opset & Versioning Concepts
The moving parts behind ONNX's cross-version and cross-tool compatibility.
- opset_version- per-domain operator set version; mismatched opsets between exporter and runtime are the most common conversion failure
- onnx.version_converter.convert_version(model, target)- upgrades or downgrades a model between opset versions
- onnx.compose.merge_models(a, b, io_map)- stitches two ONNX graphs together (e.g. preprocessing + model) into one file
- onnx.shape_inference.infer_shapes(model)- statically propagates tensor shapes through the graph for debugging
- custom domain ops- vendor ops (com.microsoft, ai.onnx.contrib) outside the core ai.onnx domain, needed for fused/attention kernels
- IR version- the ONNX file-format version, distinct from and independent of the opset version
Pro Tip
Always set dynamic_axes for the batch dimension on export — a model hardcoded to batch size 1 will silently fail or require re-export the moment you need to batch requests in production.
Was this cheat sheet helpful?
Explore Topics
#ONNXModelInterchange#ONNXModelInterchangeCheatSheet#DataScience#Intermediate#Export#PyTorch#Model#ONNX#Networking#MachineLearning#CheatSheet#SkillVeris
