liveBook · Manning

Online quantization

As explained in Build a DeepSeek Model (From Scratch)

Online quantization calculates quantization scaling factors from the current tensor or batch during processing rather than from historical statistics.

Online quantization derives a fresh scaling factor from the current tensor or batch instead of historical statistics. This addresses rapidly changing data distributions: stale scales can cause overflow when values grow or underflow and precision loss when values shrink. section 5.6.5

The implementation follows a short runtime calibration pass:

  • Inspect the current activations or weights and find their maximum absolute value. section 5.6.5
  • Derive the scaling factor from that online maximum. section 5.6.5
  • Quantize the current tensor using the freshly calculated factor. section 5.6.5
  • Pay a small maximum-finding overhead to obtain more accurate scaling before the expensive matrix multiplications. section 5.6.5

The result is a scale tailored to the exact dynamic range being processed, avoiding instability from stale historical information. Online quantization functions as one component of a broader FP8 design alongside mixed precision, fine-grained quantization, higher-precision accumulation, and mantissa-focused format choices. section 5.6.5 section 5.7

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains the failure mode, three-step runtime workflow, and computational trade-off without detailing kernel implementation.
Examples
Uses the current batch maximum absolute value and its derived scaling factor as the central implementation example.
What it assumes
Assumes foundational knowledge of quantization and numerical formats before introducing dynamic scaling.
What you can do after
Apply real-time scaling to current tensors while evaluating the inexpensive calibration pass against matrix-multiplication cost.
How it connects
Connects scaling factors with FP8 quantization, activations, weights, fine-grained quantization, and accumulation precision.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.