An imatrix records per-tensor activation magnitudes by running a calibration corpus through an F16 model. llama-quantize reads the sidecar to weight its codebook fit toward tensors that matter most to model loss. section 10.3.3
IQ-quants use an imatrix; k-quants can use one but often work well without it. The estimates guide quantization, not fine-tuning. section 10.3.3
Calibration and measured trade-off
A 1500-word Wikitext-2 slice supplied three 512-token chunks for calibration. In one comparison, IQ4_XS was 3.45 GB with PPL 5.36, versus Q4_K_M at 3.89 GB and PPL 5.43; the 1500-word evaluation's ±0.47 standard-error band is too wide to distinguish those scores statistically. section 10.3.3 section 10.3.4
Listing 10.6 imatrix calibration (one-time, IQ-quants only)
./build/bin/llama-imatrix \
-m _models/Llama-2-7b-hf-F16.gguf \
-f _models/imatrix_calibration.txt \
-o _models/imatrix.gguf \
-c 512 --chunks 10 \
--n-gpu-layers 0 CPU-only one-shot calibration
# Wall-clock: 89.2s. Output: imatrix.gguf, 4.4 MB.
# Run logged: "computing over 3 chunks, n_ctx=512, batch_size=2048, n_seq=4"
Use the one-time CPU-only llama-imatrix command as a calibration workflow example. section 10.3.3
