liveBook · Manning

Bit precision

As explained in Quantization and Fast Inference

The number of bits used to represent each numerical value.

Bit precision is the number of bits used to represent each numerical value. In inference, it is an energy and memory decision as well as an accuracy decision: more bits increase the data stored and moved, potentially limiting throughput before arithmetic does. Reducing a value’s representation from 8 bits to 4 bits halves its raw payload, excluding quantization metadata and other overhead. section 1.1

Choose precision together with the tensor to quantize and the granularity of its scale factors. Weights primarily affect model storage and traffic; quantizing activations can also speed computation but requires calibration data. For transformers, KV-cache traffic grows with context length, making its precision increasingly consequential. section 3.6 section 1.1

A practical configuration process:

  • Select targets: quantize linear and convolutional weights, consider activations for compute speedup, and consider the KV cache as context length grows. Embeddings, classification heads, and biases may need higher precision. section 3.6
  • Select bit-width and granularity together. INT8 weights commonly use per-channel scales; INT4 weights commonly use groups of 128. INT2 weights may need smaller groups and higher-precision outliers. section 3.6
  • Check model sensitivity and accuracy tolerance. Transformer activation outliers and generation errors can make aggressive precision harder to use than in less sensitive classification workloads. section 3.6 section 4.1
  • Verify runtime support before committing. If a chosen granularity is unsupported, fall back to a supported, coarser scheme and validate accuracy and latency. section 3.6
Table 3.8 Recommended granularity by target and bit-width
Target
INT8
INT4
INT2
Weights (Linear/Conv)
Per-channel (output axis)
Group-128 (input axis)
Group-32 (+ outlier channels in FP16)
Weights (Embeddings)
Per-row or Per-channel
Group-128
Group-32 + Mixed precision
Activations
Per-tensor (CNN) or Per-token (Transformer)
Per-token (dynamic)
Not recommended
KV Cache Keys
Per-channel
Per-channel
Research only
KV Cache Values
Per-token
Per-token
Research only

Compare recommended scale granularity for weights, activations, and KV-cache keys and values at INT8, INT4, and INT2. section 3.6

Start with post-training quantization when its accuracy is sufficient. Its viability depends on architecture, target bit-width, model size and redundancy, and tolerated degradation. Quantization-aware training can recover accuracy when post-training quantization falls short, but requires retraining, tuning, and additional engineering. section 4.1

Two precision decisions

A large CNN targeting INT8 with a 2% accuracy tolerance is a favorable case for post-training quantization. A small attention-heavy model targeting INT4 for medical imaging, with little tolerance for accuracy loss, is much more likely to require quantization-aware training. section 4.1

Table 3.9 Runtime support matrix
Runtime
Per-channel weights
Group Quantization
Per-token Activations
TensorRT
✓ Supported
✓ Supported
✓ Supported
ONNX Runtime
✓ Supported
✓ Supported
✓ Supported
llama.cpp
✓ Via GGUF
✓ Native (Q4_K, Q5_K)
N/A (weights-only)
TFLite
✓ Supported
Limited
Per-tensor only
PyTorch (eager)
✓ Supported
✓ Supported
✓ Supported

Check runtime support for per-channel weights, group quantization, and per-token activations before selecting a precision scheme. section 3.6

How Quantization and Fast Inference covers it

How deep it goes
Provides a target-and-granularity checklist and a PTQ-versus-QAT decision framework, including hardware and accuracy checks.
Examples
Contrasts a large CNN targeting INT8 with a small attention-heavy medical imaging model targeting INT4.
What you can do after
Evaluate precision choices against model sensitivity, accuracy tolerance, latency, and runtime support.
How it connects
Connects bit precision to quantization granularity, calibration, KV cache, post-training quantization, and quantization-aware training.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.