liveBook · Manning

Q4_K_M format

As explained in Vibe Engineering

Q4_K_M is a model quantization configuration that stores weights at approximately four-bit precision using a specified K-family quantization variant.

Q4_K_M is the practical default for local developer inference: it reduces parameter precision to roughly four bits, shrinking memory use while preserving enough quality for most coding tasks. The shortcut fails when lower precision undermines coherent multi-line generation, so function-length code should stay at Q4_K_M or above. section 9.3.5 section 9.3.1

Use these decision rules when choosing a local model download:

  • Choose Q4_K_M for most developer workloads, especially tab completion, classification, inline edits, and test scaffolding. section 9.3.5
  • Expect roughly 8.5 GB for a 14B model, at least 12 GB of RAM, approximately 94% quality, and 1.8× reference speed. section 9.3.5
  • Prefer Q6_K when 64 GB or more leaves ample headroom; its quality advantage matters more than its additional model size. section 9.3.5
  • Avoid going below Q4_K_M for multi-line functions or test suites, where inconsistent names, broken control flow, and forgotten constraints become noticeable. section 9.3.5
  • Check total memory rather than model size alone: unified-memory Macs share capacity with the IDE and development tools, while discrete GPUs impose a VRAM limit. section 9.3.2 section 9.3.1

A measured default

An on-premise developer team compared Q4_K_M with Q6_K across tab completion, inline edits, and test scaffolding. Acceptance differed by less than 2% across 500 completions, so the team selected Q4_K_M to preserve RAM for the IDE and Docker containers. section 9.3.5

Table 9.5 Quantization levels for local coding - what to download
Level
Size (14B model)
RAM Needed
Quality*
Speed
Recommendation
FP16
28 GB
32+ GB
100%
1.0x
Only for eval
Q8
15 GB
20+ GB
~99%
1.3x
If RAM allows
Q6_K
12 GB
16+ GB
~97%
1.5x
Quality-first
Q4_K_M
8.5 GB
12+ GB
~94%
1.8x
Sweet spot ←
Q4_K_S
8.0 GB
11+ GB
~93%
1.9x
Slightly smaller
Q3_K_M
6.5 GB
9+ GB
~88%
2.0x
RAM-constrained
Q2_K
5.0 GB
7+ GB
~78%
2.2x
Last resort

Compare quantization levels by model size, RAM requirement, quality, speed, and recommended use. section 9.3.5

How Vibe Engineering covers it

How deep it goes
Explains bit reduction, memory tradeoffs, benchmark differences, and generation failures without detailing the K-family algorithm internals.
Examples
Uses a 14B model table, a 500-completion Q4_K_M versus Q6_K comparison, and coding tasks including completion and test scaffolding.
What it assumes
Assumes basic model and hardware concepts while defining parameter precision, bit reduction, and memory-quality tradeoffs from zero.
What you can do after
Evaluate local model downloads and select Q4_K_M, Q6_K, or lower levels against available memory and workload demands.
How it connects
Connects quantization with local inference, model weights, RAM, VRAM, Q6_K, Q4_K_S, and multi-line code generation.

Also explained in

Compare how 2 books explain q4_K_M format →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.