K-quantization compresses floating-point model weights into low-bit values while preserving their relative magnitudes. Its defining mechanism is a hierarchy: a 256-weight superblock uses one FP16 scale for overall magnitude and 16 INT8 sub-scales, one for each 16-weight chunk, to capture local variation. This gives weights fewer raw bits while the scales handle dynamic range. section 10.2.1 section 10.6
Choose a recipe based on how much precision to allocate across the model:
- S variants quantize every weight uniformly to the nominal K-quant. section 10.2.1
- M and L variants raise the precision of selected sensitive tensors, typically attn_v, ffn_down, and the output projection. In the described Q4_K_M file, these tensors use Q6_K. section 10.2.1
- Compare effective bits per weight, not just the format label: Q3_K_M is about 3.6 bpw and retains quality that Q4_0 at 4.5 bpw loses. section 10.2.1
Figure 10.2 GGUF quantization formats by bits per weight, including per-block scale and metadata overhead. Legacy quants (dotted) span Q4_0 (4.50 bpw) through Q8_0 (8.50). K-quants (diagonal) span Q2_K (2.62) through Q6_K (6.56). Bars show each base format's bpw; the recipe-level M-variants (Q3_K_M, Q4_K_M, Q5_K_M) sit 0.2–0.4 bpw above their base because of bumped tensors, as 10.2.4 makes explicit. IQ-quants (cross) span IQ1_S (1.56) through IQ4_XS (4.25). Importance-matrix calibration pushes below the k-quant floor. Reference dashes mark INT4 and INT8.

Compare K-quant formats by effective bits per weight, including scale and metadata overhead; recipe-level M variants cost additional bits for promoted tensors. section 10.2.1
K-quants are GGUF formats used with llama.cpp, not a layout that round-trips through ONNX. In the conversion workflow, llama-quantize rewrites an FP16 GGUF into a target such as Q4_K_M. For CPU decode, the format is a major throughput lever because each generated token reads the model's weight footprint. section 10.6 section 10.0
