liveBook · Manning

K-quantization

As explained in Quantization and Fast Inference

K-quant is a family of llama.cpp low-bit quantization schemes that store model values in structured blocks with associated scaling information.

K-quantization compresses floating-point model weights into low-bit values while preserving their relative magnitudes. Its defining mechanism is a hierarchy: a 256-weight superblock uses one FP16 scale for overall magnitude and 16 INT8 sub-scales, one for each 16-weight chunk, to capture local variation. This gives weights fewer raw bits while the scales handle dynamic range. section 10.2.1 section 10.6

Choose a recipe based on how much precision to allocate across the model:

  • S variants quantize every weight uniformly to the nominal K-quant. section 10.2.1
  • M and L variants raise the precision of selected sensitive tensors, typically attn_v, ffn_down, and the output projection. In the described Q4_K_M file, these tensors use Q6_K. section 10.2.1
  • Compare effective bits per weight, not just the format label: Q3_K_M is about 3.6 bpw and retains quality that Q4_0 at 4.5 bpw loses. section 10.2.1
Figure 10.2 GGUF quantization formats by bits per weight, including per-block scale and metadata overhead. Legacy quants (dotted) span Q4_0 (4.50 bpw) through Q8_0 (8.50). K-quants (diagonal) span Q2_K (2.62) through Q6_K (6.56). Bars show each base format's bpw; the recipe-level M-variants (Q3_K_M, Q4_K_M, Q5_K_M) sit 0.2–0.4 bpw above their base because of bumped tensors, as 10.2.4 makes explicit. IQ-quants (cross) span IQ1_S (1.56) through IQ4_XS (4.25). Importance-matrix calibration pushes below the k-quant floor. Reference dashes mark INT4 and INT8.

Compare K-quant formats by effective bits per weight, including scale and metadata overhead; recipe-level M variants cost additional bits for promoted tensors. section 10.2.1

K-quants are GGUF formats used with llama.cpp, not a layout that round-trips through ONNX. In the conversion workflow, llama-quantize rewrites an FP16 GGUF into a target such as Q4_K_M. For CPU decode, the format is a major throughput lever because each generated token reads the model's weight footprint. section 10.6 section 10.0

How Quantization and Fast Inference covers it

How deep it goes
Explains 256-weight superblocks, two-level scaling, and recipe variants, then compares effective bits and quality.
Examples
Compares Q3_K_M with Q4_0 and describes Q4_K_M promoting selected tensors to Q6_K.
What you can do after
Apply K-quantization by converting an FP16 GGUF with llama-quantize to a target format such as Q4_K_M.
How it connects
Connects GGUF, legacy quantization, IQ-quants, and FP16 weight conversion.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.