Bit precision is the number of bits used to represent each numerical value. In inference, it is an energy and memory decision as well as an accuracy decision: more bits increase the data stored and moved, potentially limiting throughput before arithmetic does. Reducing a value’s representation from 8 bits to 4 bits halves its raw payload, excluding quantization metadata and other overhead. section 1.1
Choose precision together with the tensor to quantize and the granularity of its scale factors. Weights primarily affect model storage and traffic; quantizing activations can also speed computation but requires calibration data. For transformers, KV-cache traffic grows with context length, making its precision increasingly consequential. section 3.6 section 1.1
A practical configuration process:
- Select targets: quantize linear and convolutional weights, consider activations for compute speedup, and consider the KV cache as context length grows. Embeddings, classification heads, and biases may need higher precision. section 3.6
- Select bit-width and granularity together. INT8 weights commonly use per-channel scales; INT4 weights commonly use groups of 128. INT2 weights may need smaller groups and higher-precision outliers. section 3.6
- Check model sensitivity and accuracy tolerance. Transformer activation outliers and generation errors can make aggressive precision harder to use than in less sensitive classification workloads. section 3.6 section 4.1
- Verify runtime support before committing. If a chosen granularity is unsupported, fall back to a supported, coarser scheme and validate accuracy and latency. section 3.6
Table 3.8 Recommended granularity by target and bit-width
|
Target
|
INT8
|
INT4
|
INT2
|
|
Weights (Linear/Conv)
|
Per-channel (output axis)
|
Group-128 (input axis)
|
Group-32 (+ outlier channels in FP16)
|
|
Weights (Embeddings)
|
Per-row or Per-channel
|
Group-128
|
Group-32 + Mixed precision
|
|
Activations
|
Per-tensor (CNN) or Per-token (Transformer)
|
Per-token (dynamic)
|
Not recommended
|
|
KV Cache Keys
|
Per-channel
|
Per-channel
|
Research only
|
|
KV Cache Values
|
Per-token
|
Per-token
|
Research only
|
Compare recommended scale granularity for weights, activations, and KV-cache keys and values at INT8, INT4, and INT2. section 3.6
Start with post-training quantization when its accuracy is sufficient. Its viability depends on architecture, target bit-width, model size and redundancy, and tolerated degradation. Quantization-aware training can recover accuracy when post-training quantization falls short, but requires retraining, tuning, and additional engineering. section 4.1
Two precision decisions
A large CNN targeting INT8 with a 2% accuracy tolerance is a favorable case for post-training quantization. A small attention-heavy model targeting INT4 for medical imaging, with little tolerance for accuracy loss, is much more likely to require quantization-aware training. section 4.1
Table 3.9 Runtime support matrix
|
Runtime
|
Per-channel weights
|
Group Quantization
|
Per-token Activations
|
|
TensorRT
|
✓ Supported
|
✓ Supported
|
✓ Supported
|
|
ONNX Runtime
|
✓ Supported
|
✓ Supported
|
✓ Supported
|
|
llama.cpp
|
✓ Via GGUF
|
✓ Native (Q4_K, Q5_K)
|
N/A (weights-only)
|
|
TFLite
|
✓ Supported
|
Limited
|
Per-tensor only
|
|
PyTorch (eager)
|
✓ Supported
|
✓ Supported
|
✓ Supported
|
Check runtime support for per-channel weights, group quantization, and per-token activations before selecting a precision scheme. section 3.6
