ONNX, or Open Neural Network Exchange, is a vendor-neutral format for describing neural-network and machine-learning computations. It lets a model move beyond its original framework and run with optimized processors, specialized hardware, cloud services, or applications that do not use Python. section 17.0 section 1.4.1
A deep-learning model can be treated as a program built from operations such as matrix multiplication, convolution, ReLU, and tanh. ONNX serializes those operations and their parameters so an ONNX-compatible runtime can re-execute them. section 17.2.1
A basic ONNX deployment workflow is:
- Run the model with a dummy input whose shape and type match the expected input; the values themselves are not important. section 17.2.1
- Call torch.onnx.export to trace the model’s computations and write an ONNX file. section 17.2.1
- Load the file with a compatible runtime such as ONNX Runtime and execute it with NumPy input data. section 17.2.1
- Check that the model’s operations are supported by both the ONNX standard and the target runtime. section 17.2.1
onnx_model_path = "simple_model.onnx"
dummy_input = torch.randn(1, 10)
torch.onnx.export( #1
model,
dummy_input,
onnx_model_path
)
This short export listing shows the required model, correctly shaped dummy input, and output path. section 17.2.1
import onnxruntime
sess = onnxruntime.InferenceSession("simple_model.onnx") #1
batch = dummy_input.numpy()
input_name = sess.get_inputs()[0].name
pred_onnx, = sess.run(None, {input_name: batch})
This inference listing loads the exported file with ONNX Runtime, discovers its input name, and runs a prediction. section 17.2.1
