Q4_K_M format
Q4_K_M is a model-weight quantization format that uses predominantly four-bit K-family quantization, with some values represented at higher precision.
Quantization and Fast Inference
Q4_K_M is a llama.cpp quantization format that stores model weights primarily using four-bit K-quant blocks.
- How it is used
- Validates converted GGUF artifacts and serves as a deployment fallback when runtimes cannot read 16-entry codebooks.
- How deep it goes
- Reports its enum code, measured size and perplexity, and a representative Q4_K kernel path.
- Examples
- Uses Wikitext-2 size and perplexity measurements, CPU throughput tests, and an Apollo-prompt generation check.
- What you can do after
- Evaluate size, perplexity, fluency, and decode-throughput tradeoffs, then choose Q4_K_M as a compatibility fallback when needed.
Read how Quantization and Fast Inference explains q4_K_M format →
Vibe Engineering
Q4_K_M is a model quantization configuration that stores weights at approximately four-bit precision using a specified K-family quantization variant.
- How it is used
- Guides local model-download choices by balancing coding quality, memory consumption, and generation speed.
- How deep it goes
- Explains bit reduction, memory tradeoffs, benchmark differences, and generation failures without detailing the K-family algorithm internals.
- Examples
- Uses a 14B model table, a 500-completion Q4_K_M versus Q6_K comparison, and coding tasks including completion and test scaffolding.
- What you can do after
- Evaluate local model downloads and select Q4_K_M, Q6_K, or lower levels against available memory and workload demands.
Also covered in
What to know first
- AVX-512AVX-512 is a family of x86 instruction-set extensions that supports SIMD operations on vector registers up to 512 bits wide.
- memoryMemory is a storage capability or mechanism for retaining information or computational state for later access or use.
Related concepts
- FP32uses · FP32 is a 32-bit single-precision floating-point format for representing approximate real numbers.
- IQ4_XScontrasts with · IQ4_XS is a llama.cpp importance-matrix quantization format using approximately four bits per weight and codebook-based reconstruction.
- Q6_Kuses · Q6_K is a K-family model quantization format that encodes weights using six-bit values and associated scales.
- K-quantizationkind of · K-quantization is a family of fixed-heuristic quantization methods that assigns nominal bit widths and selectively higher-precision formats to particular tensors.
- quantizationkind of · Model quantization represents model parameters or computations with reduced-precision numerical values, often to lower storage or computational requirements.
Questions about q4_K_M format
What is q4_K_M format?
Q4_K_M is a model-weight quantization format that uses predominantly four-bit K-family quantization, with some values represented at higher precision.
Which Manning books explain q4_K_M format?
Quantization and Fast Inference and Vibe Engineering.
What should I know before learning q4_K_M format?
Manning books that teach q4_K_M format usually introduce AVX-512 and memory first.
What is the difference between q4_K_M format and IQ4_XS?
Q4_K_M is a model-weight quantization format that uses predominantly four-bit K-family quantization, with some values represented at higher precision. IQ4_XS is a llama.cpp importance-matrix quantization format using approximately four bits per weight and codebook-based reconstruction.
What is q4_K_M format a kind of?
Q4_K_M format is a kind of K-quantization. K-quantization is a family of fixed-heuristic quantization methods that assigns nominal bit widths and selectively higher-precision formats to particular tensors.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.

