liveBook · Manning

Granularity

As explained in Quantization and Fast Inference

The level of tensor partitioning that determines which values share quantization parameters such as a scale factor.

Granularity determines which tensor values share a quantization scale. Finer partitions can reduce quantization error but require more scale factors; per-channel quantization yields 3–9× lower MSE than per-tensor, while group-128 is about 2× better than per-channel with roughly 3% overhead. section 3.7

Choose granularity for each tensor target and bit-width:

  • Weights: use per-channel scaling along the output axis for INT8, group-128 along the input axis for INT4, and group-32 with FP16 outlier channels for INT2. section 3.6
  • Activations: use per-tensor scaling for CNNs or per-token scaling for transformers at INT8; use dynamic per-token scaling at INT4. INT2 activations are not recommended. section 3.6
  • KV cache: quantize keys per-channel and values per-token; INT2 cache quantization remains research-only. section 3.6

Align groups with the contiguous memory axis for coalesced loads. Check runtime support before deployment; if the chosen scheme is unsupported, fall back to a coarser granularity. section 3.7 section 3.6

Granularity changes the comparison

On OPT-6.7B fc1 layers, FP4 block-16 achieves 2.65×10⁻⁶ reconstruction MSE versus 4.56×10⁻⁶ for INT4 group-128. Block-16 uses eight times as many scale factors, so the result reflects both finer scaling and FP4’s spacing—not format alone. section 8.3.3

Figure 3.12 End-to-end quantization decision tree, read top-to-bottom. Phase 1 (top): select which tensors to quantize—weights are always included, activations add compute speedup, KV cache is context-length dependent. Phase 2 (middle): for each target, select granularity using bit-width as the primary fork—INT8 paths favor per-channel/per-token, INT4 paths favor group-128. Phase 3 (bottom): architecture-specific overrides (e.g., first/last layers in FP16). All paths converge on the hardware compatibility gate (right)—if your runtime doesn't support the chosen scheme, fall back to the next coarser granularity.

Use the decision tree to follow target selection, bit-width-based granularity, architecture adjustments, and the hardware support check. section 3.6

Table 3.8 Recommended granularity by target and bit-width
Target
INT8
INT4
INT2
Weights (Linear/Conv)
Per-channel (output axis)
Group-128 (input axis)
Group-32 (+ outlier channels in FP16)
Weights (Embeddings)
Per-row or Per-channel
Group-128
Group-32 + Mixed precision
Activations
Per-tensor (CNN) or Per-token (Transformer)
Per-token (dynamic)
Not recommended
KV Cache Keys
Per-channel
Per-channel
Research only
KV Cache Values
Per-token
Per-token
Research only

Compare recommended granularity by tensor target and bit-width. section 3.6

How Quantization and Fast Inference covers it

How deep it goes
Compares per-tensor, per-channel, and grouped scaling through MSE, scale-factor overhead, and memory-layout tradeoffs.
Examples
Compares INT8 per-channel with per-tensor scaling and FP4 block-16 with INT4 group-128 on weight reconstruction error.
What you can do after
Configure tensor-specific scaling schemes by bit-width, then verify runtime support and memory-layout alignment.
How it connects
Connects quantization granularity to bit-width, activation calibration, quantization-aware training, and hardware compatibility.

Also explained in

Compare how 13 books explain granularity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.