liveBook · Manning

Perplexity

As explained in Quantization and Fast Inference

Perplexity is a language-model metric measuring how well a probability model predicts a sequence, with lower values indicating better predictive performance.

Perplexity measures how well a probability model predicts a sequence; lower values indicate better predictive performance. For quantized models, use it to compare end-to-end quality, not as a substitute for weight-reconstruction error or fluency checks. section 8.2.3 section 10.3.4

Comparing quantized variants

On a 1,500-word evaluation slice, IQ4_XS measured PPL 5.36 and Q4_K_M 5.43. The reported standard error was ±0.47, so that run alone cannot establish a reliable difference between the variants. section 10.3.4

Figure 10.5 Wikitext-2 perplexity (4 chunks × 512 tokens, CPU-only) versus on-disk size. F16 (12.85 GB, PPL 5.27) anchors the lossless baseline. F16 → Q8_0 → Q5_K_M is a clean staircase down. Q4_K_M sits to the upper side of Q5_K_M as expected. IQ4_XS sits to the lower-left of Q4_K_M—smaller and lower perplexity at the same bit tier.

Compare perplexity with model size across quantization variants; F16 provides the baseline, while IQ4_XS is smaller and lower in perplexity than Q4_K_M in this evaluation. section 10.3.4

How Quantization and Fast Inference covers it

How deep it goes
Reports perplexity measurements and uncertainty bands for quantized variants without detailing the metric's computation.
Examples
Compares Q4_K_M and IQ4_XS perplexity, including a short-slice estimate whose uncertainty exceeds their measured gap.
What you can do after
Evaluate quantized model variants using perplexity while accounting for sample uncertainty and complementary fluency checks.
How it connects
Relates quantization quality to weight reconstruction error, fluency, and Wikitext-2 evaluation.

Also explained in

Compare how 9 books explain perplexity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.