8-bit precision reduces model weights or activations from higher-precision formats to 8-bit values, shrinking memory use and potentially accelerating inference while introducing quantization error. 8-bit values can store about four times as many numbers as 32-bit floats in the same memory. section 6.2
Use 8-bit quantization through these practical paths:
- Apply post-training quantization after training; use quantization-aware training during training when the workflow requires it. section 6.2
- Convert transformer models to ONNX with PyTorch or Transformers, then quantize them through ONNX Runtime. section 6.3
- Use Optimum for higher-level ONNX export and hardware-oriented optimization; configure Transformers directly when Optimum lacks model support. section 6.3 section 7.2.2
- Quantize weights alone for simpler workflows, or quantize both weights and activations when targeting W8A8 execution. section 9.2
Quantizing a sequence classifier
Export bert-base-uncased to ONNX, apply ONNX Runtime dynamic quantization with QInt8 weights, reload the quantized model through Optimum, and compare accuracy and latency against the unquantized pipeline. section 6.3
Scaling 8-bit quantization to larger models
SmoothQuant transforms activation magnitudes so an OPT model can use 8-bit weights and activations without additional training. Reported experiments show up to 1.56× speedup and 50% lower memory use with minimal accuracy loss. section 9.2
Figure 9.3 A high-level explanation of SmoothQuant
Use the high-level SmoothQuant figure to inspect how scaling shifts quantization difficulty from activations toward weights. section 9.2
