Granularity determines which tensor values share a quantization scale. Finer partitions can reduce quantization error but require more scale factors; per-channel quantization yields 3–9× lower MSE than per-tensor, while group-128 is about 2× better than per-channel with roughly 3% overhead. section 3.7
Choose granularity for each tensor target and bit-width:
- Weights: use per-channel scaling along the output axis for INT8, group-128 along the input axis for INT4, and group-32 with FP16 outlier channels for INT2. section 3.6
- Activations: use per-tensor scaling for CNNs or per-token scaling for transformers at INT8; use dynamic per-token scaling at INT4. INT2 activations are not recommended. section 3.6
- KV cache: quantize keys per-channel and values per-token; INT2 cache quantization remains research-only. section 3.6
Align groups with the contiguous memory axis for coalesced loads. Check runtime support before deployment; if the chosen scheme is unsupported, fall back to a coarser granularity. section 3.7 section 3.6
Granularity changes the comparison
On OPT-6.7B fc1 layers, FP4 block-16 achieves 2.65×10⁻⁶ reconstruction MSE versus 4.56×10⁻⁶ for INT4 group-128. Block-16 uses eight times as many scale factors, so the result reflects both finer scaling and FP4’s spacing—not format alone. section 8.3.3
Figure 3.12 End-to-end quantization decision tree, read top-to-bottom. Phase 1 (top): select which tensors to quantize—weights are always included, activations add compute speedup, KV cache is context-length dependent. Phase 2 (middle): for each target, select granularity using bit-width as the primary fork—INT8 paths favor per-channel/per-token, INT4 paths favor group-128. Phase 3 (bottom): architecture-specific overrides (e.g., first/last layers in FP16). All paths converge on the hardware compatibility gate (right)—if your runtime doesn't support the chosen scheme, fall back to the next coarser granularity.

Use the decision tree to follow target selection, bit-width-based granularity, architecture adjustments, and the hardware support check. section 3.6
Table 3.8 Recommended granularity by target and bit-width
|
Target
|
INT8
|
INT4
|
INT2
|
|
Weights (Linear/Conv)
|
Per-channel (output axis)
|
Group-128 (input axis)
|
Group-32 (+ outlier channels in FP16)
|
|
Weights (Embeddings)
|
Per-row or Per-channel
|
Group-128
|
Group-32 + Mixed precision
|
|
Activations
|
Per-tensor (CNN) or Per-token (Transformer)
|
Per-token (dynamic)
|
Not recommended
|
|
KV Cache Keys
|
Per-channel
|
Per-channel
|
Research only
|
|
KV Cache Values
|
Per-token
|
Per-token
|
Research only
|
Compare recommended granularity by tensor target and bit-width. section 3.6












