liveBook · Manning

Int4 quantization

As explained in Quantization and Fast Inference

A 4-bit integer representation used for low-precision numerical computation and storage.

INT4 stores weights using four-bit integers, reducing memory use and latency at the cost of a measured quality drop. Symmetric INT4 spaces its positive values uniformly using a scale factor. section 8.3.1 section 12.4

Decide whether INT4 fits the model and deployment:

  • Assess architecture, model size, target bit-width, and acceptable accuracy loss when judging whether post-training quantization is sufficient. section 4.1
  • Validate the deployed runtime: compare only formats actually measured, rather than treating configured alternatives as tested results. section 12.3

INT4 and FP4 level spacing

From 0 to 2, INT4 has 3 non-negative values while FP4 has 5. Across 0 to 6, both have 8; FP4 concentrates more levels near zero. section 8.3.1

Figure 8.7 Representable values in [0, 6]. FP4 E2M1 and INT4 symmetric each have 8 non-negative values (including zero), but FP4 clusters them toward the origin while INT4 spaces them uniformly. FP8 E4M3 (bottom row) packs 78 values into the same interval—roughly 10× the resolution of either 4-bit format.

Compare how symmetric INT4 spaces values uniformly while FP4 clusters them toward zero. section 8.3.1

How Quantization and Fast Inference covers it

How deep it goes
Explains scaled, uniformly spaced INT4 values and contrasts their density with FP4.
Examples
Compares INT4 and FP4 value density over [0, 2] and [0, 6], and considers INT4 in model-suitability decisions.
What you can do after
Evaluate whether aggressive INT4 post-training quantization fits a model's architecture and accuracy tolerance.
How it connects
Connects INT4 to post-training quantization, calibration, quantization-aware training, and FP4.

Also explained in

Compare how 3 books explain int4 quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.