liveBook · Manning

Q4_K_M format

Q4_K_M is a model-weight quantization format that uses predominantly four-bit K-family quantization, with some values represented at higher precision.

How 2 Manning books explain it

Quantization and Fast Inference

Q4_K_M is a llama.cpp quantization format that stores model weights primarily using four-bit K-quant blocks.

How it is used
Validates converted GGUF artifacts and serves as a deployment fallback when runtimes cannot read 16-entry codebooks.
How deep it goes
Reports its enum code, measured size and perplexity, and a representative Q4_K kernel path.
Examples
Uses Wikitext-2 size and perplexity measurements, CPU throughput tests, and an Apollo-prompt generation check.
What you can do after
Evaluate size, perplexity, fluency, and decode-throughput tradeoffs, then choose Q4_K_M as a compatibility fallback when needed.

Read how Quantization and Fast Inference explains q4_K_M format →

Vibe Engineering

Q4_K_M is a model quantization configuration that stores weights at approximately four-bit precision using a specified K-family quantization variant.

How it is used
Guides local model-download choices by balancing coding quality, memory consumption, and generation speed.
How deep it goes
Explains bit reduction, memory tradeoffs, benchmark differences, and generation failures without detailing the K-family algorithm internals.
Examples
Uses a 14B model table, a 500-completion Q4_K_M versus Q6_K comparison, and coding tasks including completion and test scaffolding.
What you can do after
Evaluate local model downloads and select Q4_K_M, Q6_K, or lower levels against available memory and workload demands.

Read how Vibe Engineering explains q4_K_M format →

Also covered in

What to know first

Related concepts

Questions about q4_K_M format

What is q4_K_M format?

Q4_K_M is a model-weight quantization format that uses predominantly four-bit K-family quantization, with some values represented at higher precision.

Which Manning books explain q4_K_M format?

Quantization and Fast Inference and Vibe Engineering.

What should I know before learning q4_K_M format?

Manning books that teach q4_K_M format usually introduce AVX-512 and memory first.

What is the difference between q4_K_M format and IQ4_XS?

Q4_K_M is a model-weight quantization format that uses predominantly four-bit K-family quantization, with some values represented at higher precision. IQ4_XS is a llama.cpp importance-matrix quantization format using approximately four bits per weight and codebook-based reconstruction.

What is q4_K_M format a kind of?

Q4_K_M format is a kind of K-quantization. K-quantization is a family of fixed-heuristic quantization methods that assigns nominal bit widths and selectively higher-precision formats to particular tensors.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.