liveBook · Manning

Importance matrix

As explained in Quantization and Fast Inference

Numerical data representing the relative importance of model weights for guiding quantization.

An importance matrix is generated by running a small calibration corpus through the model. It steers codebook fitting toward weights that matter most for output quality. section 10.2.1

It can improve both IQ-quant and k-quant fitting. At sub-4-bit budgets, codebook fitting is too lossy without calibration, making the matrix practically inseparable from IQ-quants. section 10.2.1

Figure 10.2 GGUF quantization formats by bits per weight, including per-block scale and metadata overhead. Legacy quants (dotted) span Q4_0 (4.50 bpw) through Q8_0 (8.50). K-quants (diagonal) span Q2_K (2.62) through Q6_K (6.56). Bars show each base format's bpw; the recipe-level M-variants (Q3_K_M, Q4_K_M, Q5_K_M) sit 0.2–0.4 bpw above their base because of bumped tensors, as 10.2.4 makes explicit. IQ-quants (cross) span IQ1_S (1.56) through IQ4_XS (4.25). Importance-matrix calibration pushes below the k-quant floor. Reference dashes mark INT4 and INT8.

Compare quantization formats by bits per weight; the chart notes that importance-matrix calibration enables formats below the k-quant floor. section 10.2.1

How Quantization and Fast Inference covers it

How deep it goes
Describes generation from a small calibration corpus and its fitting role, without a numerical construction walkthrough.
Examples
Uses a small corpus run through the model as calibration input.
What you can do after
Apply calibration to steer sub-4-bit codebook fitting toward output quality.
How it connects
Links calibration to codebook fitting, IQ-quants, and k-quants.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.