liveBook · Manning

Symbolic shape inference

As explained in Quantization and Fast Inference

Symbolic shape inference determines tensor dimensions through a computational graph using symbolic expressions.

Symbolic shape inference determines tensor dimensions through a computational graph. In ONNX Runtime static quantization, resolved shapes support calibration; unresolved shapes can produce poor activation ranges and damage accuracy. section 6.4

For BERT, generic ONNX preprocessing failed to complete shape inference, and the resulting static-quantized model reached only 50.92% accuracy. The proposed correction is the transformer-aware optimizer, which resolves shapes before calibration. section 6.4

How Quantization and Fast Inference covers it

How deep it goes
Traces failed inference to unresolved tensor shapes and poor activation ranges, then identifies a transformer-aware correction.
Examples
Uses a BERT graph whose generic preprocessing failed, contrasting it with transformer-aware optimization.
What you can do after
Diagnose failed static quantization by checking preprocessing shape resolution before calibration.
How it connects
Connects ONNX Runtime preprocessing with static quantization and calibration.

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.