Quantization reduces model storage and memory pressure by rounding parameters from higher precision to fewer bits. Q8 uses eight bits, Q4 uses four; smaller files fit on constrained machines but sacrifice some accuracy. section 9.3.1
The engineering dilemma is not simply quality versus size. Memory saved by quantization can keep an IDE, browser, containers, and test runner usable, while excessive reduction can damage reasoning and coherent code generation. section 9.3.1 section 9.3.5
Choose a level according to workload and available RAM:
- Start with Q4_K_M for most developer workloads; it balances model size, quality, and memory headroom. section 9.3.5
- Prefer Q6_K when the machine has ample RAM and quality matters more than the additional model size. section 9.3.5
- Avoid going below Q4 for function-length generation or test-suite creation, where variable consistency and control-flow correctness degrade noticeably. section 9.3.5
- Treat Q2 and Q3 as constrained-memory options rather than general defaults; short tab completion may tolerate their losses better than multi-step generation. section 9.3.5
A practical download decision
A 16GB MacBook Pro running Qwen 2.5 Coder 7B at Q4_K_M can enter swap because the model shares unified memory with the development environment. On a 32GB machine, the same small-model setup is more comfortable; with 64GB or more, Q6_K becomes a stronger quality-first option. section 9.3.1 section 9.3.5
Table 9.5 Quantization levels for local coding - what to download
|
Level
|
Size (14B model)
|
RAM Needed
|
Quality*
|
Speed
|
Recommendation
|
|
FP16
|
28 GB
|
32+ GB
|
100%
|
1.0x
|
Only for eval
|
|
Q8
|
15 GB
|
20+ GB
|
~99%
|
1.3x
|
If RAM allows
|
|
Q6_K
|
12 GB
|
16+ GB
|
~97%
|
1.5x
|
Quality-first
|
|
Q4_K_M
|
8.5 GB
|
12+ GB
|
~94%
|
1.8x
|
Sweet spot ←
|
|
Q4_K_S
|
8.0 GB
|
11+ GB
|
~93%
|
1.9x
|
Slightly smaller
|
|
Q3_K_M
|
6.5 GB
|
9+ GB
|
~88%
|
2.0x
|
RAM-constrained
|
|
Q2_K
|
5.0 GB
|
7+ GB
|
~78%
|
2.2x
|
Last resort
|
Compare quantization levels by model size, RAM requirement, estimated quality, speed, and recommended use. section 9.3.5





















