liveBook · Manning

Q4_K_M format

As explained in Quantization and Fast Inference

Q4_K_M is a llama.cpp quantization format that stores model weights primarily using four-bit K-quant blocks.

Q4_K_M is a llama.cpp quantization format used in GGUF model files. Its canonical file-type code is 15; measured at 3.89 GB with perplexity 5.43, it provides a compact option whose quality and runtime behavior should be checked on the target hardware. section 10.3.4 section 10.4.1

Compare the format against deployment constraints:

  • In one Wikitext-2 comparison, IQ4_XS measured 3.45 GB and PPL 5.36, versus Q4_K_M at 3.89 GB and PPL 5.43. The test used a short slice with a ±0.47 standard-error band, so it does not establish a statistically reliable difference. section 10.3.4
  • On measured x86 kernel families, IQ4_XS decoded 6–13% faster. Q4_K_M remains a safer fallback when runtimes cannot read 16-entry codebooks, including some older GPU kernels, pre-Skylake CPUs, and embedded targets without efficient byte-permute. section 10.4.5
Figure 10.5 Wikitext-2 perplexity (4 chunks × 512 tokens, CPU-only) versus on-disk size. F16 (12.85 GB, PPL 5.27) anchors the lossless baseline. F16 → Q8_0 → Q5_K_M is a clean staircase down. Q4_K_M sits to the upper side of Q5_K_M as expected. IQ4_XS sits to the lower-left of Q4_K_M—smaller and lower perplexity at the same bit tier.

Compare on-disk size and Wikitext-2 perplexity across formats; the figure places Q4_K_M above and to the right of IQ4_XS. section 10.3.4

How Quantization and Fast Inference covers it

How deep it goes
Reports its enum code, measured size and perplexity, and a representative Q4_K kernel path.
Examples
Uses Wikitext-2 size and perplexity measurements, CPU throughput tests, and an Apollo-prompt generation check.
What you can do after
Evaluate size, perplexity, fluency, and decode-throughput tradeoffs, then choose Q4_K_M as a compatibility fallback when needed.
How it connects
Compares Q4_K_M with IQ4_XS, Q5_K_M, Q8_0, F16, GGUF, llama.cpp, and CPU quantization kernels.

Also explained in

Compare how 2 books explain q4_K_M format →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.