liveBook · Manning

Mean squared error

As explained in Quantization and Fast Inference

Mean squared error is the average squared difference between corresponding numerical values.

Mean squared error (MSE) is the average squared difference between corresponding numerical values. For quantized weights, reconstruction MSE measures how closely the quantized values match the originals; it does not alone determine end-to-end perplexity. section 8.2.3

Figure 8.5 Weight reconstruction MSE by format (log scale). INT8 per-channel achieves the lowest MSE (≈3.5×10⁻⁸ average), followed by FP8 E4M3 (≈2.1×10⁻⁷, roughly 6× worse), then FP8 E5M2 (≈8.5×10⁻⁷, roughly 4× worse than E4M3). INT8 wins here because per-channel scaling gives each output channel its own scale factor—fine-grained enough to neutralize FP8's density advantage.

The comparison shows lower reconstruction MSE for INT8 per-channel than for FP8 E4M3 or E5M2, illustrating how scaling granularity affects the measurement. section 8.2.3

How Quantization and Fast Inference covers it

How deep it goes
Measures OPT-6.7B weight reconstruction across formats and cautions that MSE alone does not establish end-to-end perplexity.
Examples
Compares OPT-6.7B fc1 weights quantized to FP8 E4M3, FP8 E5M2, and INT8 per-channel.
What you can do after
Evaluate weight reconstruction error across formats without treating it as a proxy for end-to-end perplexity.
How it connects
Connects quantization, FP8, INT8, per-channel scaling, and perplexity.

Also explained in

Compare how 17 books explain mean squared error →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.