liveBook · Manning

Bits per weight

As explained in Quantization and Fast Inference

Bits per weight is the average number of bits used to represent each model weight in a quantized format.

Bits per weight (bpw) expresses a quantized format’s average storage budget per model weight. It includes block scales and metadata, so a format’s effective bpw can differ from its nominal bit width. section 10.2.1

GGUF formats span a wide range of effective bit budgets:

  • Legacy Q4_0 uses 4.50 bpw; hierarchical Q2_K uses 2.62 bpw and Q6_K uses 6.56 bpw. section 10.2.1
  • IQ1_S reaches 1.56 bpw; recipe variants such as Q4_K_M can use 0.2–0.4 bpw more than their base format because selected tensors are promoted. section 10.2.1

Lower bpw reduces model storage and bytes read during inference, but format choice also affects quality and runtime performance. Compare measured quality and hardware behavior rather than treating bpw alone as a ranking. section 12.4 section 10.3.4

Figure 10.2 GGUF quantization formats by bits per weight, including per-block scale and metadata overhead. Legacy quants (dotted) span Q4_0 (4.50 bpw) through Q8_0 (8.50). K-quants (diagonal) span Q2_K (2.62) through Q6_K (6.56). Bars show each base format's bpw; the recipe-level M-variants (Q3_K_M, Q4_K_M, Q5_K_M) sit 0.2–0.4 bpw above their base because of bumped tensors, as 10.2.4 makes explicit. IQ-quants (cross) span IQ1_S (1.56) through IQ4_XS (4.25). Importance-matrix calibration pushes below the k-quant floor. Reference dashes mark INT4 and INT8.

Compare effective bpw ranges across legacy, K-quant, and IQ formats, including overhead and recipe-level variants. section 10.2.1

How Quantization and Fast Inference covers it

How deep it goes
Explains effective bpw with scale and metadata overhead and gives format-level values, without deriving a general calculation.
Examples
Compares Q4_0, Q2_K, Q6_K, IQ1_S, and recipe-level K-quant bit budgets.
What you can do after
Evaluate quantization formats by comparing effective bit budgets alongside measured size, quality, and hardware performance.
How it connects
Connects quantization formats to model storage, inference latency, and perplexity-based quality comparisons.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.