liveBook · Manning

Imatrix

As explained in Quantization and Fast Inference

An imatrix is a collection of per-tensor activation-magnitude estimates used to weight quantization toward tensors important to model loss.

An imatrix records per-tensor activation magnitudes by running a calibration corpus through an F16 model. llama-quantize reads the sidecar to weight its codebook fit toward tensors that matter most to model loss. section 10.3.3

IQ-quants use an imatrix; k-quants can use one but often work well without it. The estimates guide quantization, not fine-tuning. section 10.3.3

Calibration and measured trade-off

A 1500-word Wikitext-2 slice supplied three 512-token chunks for calibration. In one comparison, IQ4_XS was 3.45 GB with PPL 5.36, versus Q4_K_M at 3.89 GB and PPL 5.43; the 1500-word evaluation's ±0.47 standard-error band is too wide to distinguish those scores statistically. section 10.3.3 section 10.3.4

Listing 10.6 imatrix calibration (one-time, IQ-quants only)
./build/bin/llama-imatrix \
    -m _models/Llama-2-7b-hf-F16.gguf \
    -f _models/imatrix_calibration.txt \
    -o _models/imatrix.gguf \
    -c 512 --chunks 10 \
    --n-gpu-layers 0                       CPU-only one-shot calibration
# Wall-clock: 89.2s. Output: imatrix.gguf, 4.4 MB.
# Run logged: "computing over 3 chunks, n_ctx=512, batch_size=2048, n_seq=4"
 

Use the one-time CPU-only llama-imatrix command as a calibration workflow example. section 10.3.3

How Quantization and Fast Inference covers it

How deep it goes
Explains corpus calibration and downstream use, but stops short of fine-tuning.
Examples
Uses a Wikitext-2 calibration slice and compares IQ4_XS with Q4_K_M on size and perplexity.
What it assumes
Assumes access to an F16 model, a calibration corpus, and llama.cpp tools.
What you can do after
Apply it via llama-imatrix calibration and llama-quantize to produce an imatrix-guided quantization.
How it connects
Connects quantization with IQ-quants, k-quants, GGUF, and model perplexity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.