liveBook · Manning

Quantization

As explained in Build a DeepSeek Model (From Scratch)

Quantization is the process of representing higher-precision numerical values with lower-precision values, typically by scaling and rounding them.

Quantization represents higher-precision numerical values with lower-bit values, trading limited per-value precision for substantially lower memory use and computational cost. For large language models, this trade makes large-scale training and inference more affordable while often preserving overall model performance. section 5.5 section 5.5.1

A basic symmetric quantization pass follows these steps:

  • Find the maximum absolute value α in the input vector or tensor. section 5.5.4
  • Map the range −α to +α onto a target range such as −127 to 127, using scale = 127 / α. section 5.5.4
  • Multiply each value by the scale and round it; for example, −7.59 becomes −89 when α = 10.8. section 5.5.4
  • Dequantize by dividing the integer representation by the same scale, recovering an approximation of the original value. section 5.5.4

Modern low-precision training uses more than one precision. Core computations can use FP8, while master weights and optimizer states remain in higher precision for numerical stability. Fine-grained quantization assigns separate scales to small activation and weight blocks, reducing the effect of outliers. Intermediate results can accumulate in FP32, and online scaling can derive factors from the current batch rather than historical data. section 5.7

Memory trade-off for a 70-billion-parameter model

Quantizing weights from 32-bit to 16-bit reduces the stated storage requirement from 280 GB to 140 GB. Moving to lower precision reduces data movement and can allow the model to run on hardware with less memory, at the cost of some numerical precision. section 5.5.2

Figure 5.21 The scaling process for quantization. The original range of the FP32 tensor is mapped to the target range of the INT8 format.

Visualizes how an FP32 tensor range maps onto the INT8 target range during scaling. section 5.5.4

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains definition, memory trade-offs, scalar scaling, dequantization, and advanced mixed-precision techniques without detailing a complete training implementation.
Examples
Uses FP32-to-FP16 comparisons, an eight-color image analogy, a 70-billion-parameter memory calculation, and an FP32-to-INT8 scaling example.
What it assumes
Builds from numerical values, parameters, tensors, bit-widths, and scaling before introducing fine-grained and mixed-precision strategies.
What you can do after
Apply quantization by calculating tensor scales, rounding values into a target range, dequantizing results, and evaluating memory–precision trade-offs.
How it connects
Connects quantization with FP8, mixed-precision training, fine-grained quantization, online quantization, and accumulation precision.

Also explained in

Compare how 22 books explain quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.