liveBook · Manning

Scalar quantization

As explained in Build a DeepSeek Model (From Scratch)

Standard quantization maps an entire tensor to a lower-precision numerical range using one shared scaling factor.

Scalar quantization converts a high-precision tensor into a lower-precision numerical range using one shared scale. It finds the maximum absolute value α, computes a scale such as s = 127/α, multiplies each value by s, and rounds. Dequantization divides the integer values by s to approximate the originals. section 5.6.2

The shared scale makes the method simple but leaves it highly sensitive to outliers. A single large activation consumes most of the target range, reducing the precision available to ordinary values across the tensor. section 5.6.2

Outlier-dominated vector

For [2.0, 3.0, 500.0] quantized to signed 8-bit integers, α = 500 and s = 0.254. Rounding produces [1, 1, 127], erasing the distinction between 2.0 and 3.0. Dequantization returns approximately [3.94, 3.94, 500.0], preserving the outlier while badly distorting the smaller values. section 5.6.2

Figure 5.27 Fine-Grained Quantization for an activation vector. The vector is partitioned into smaller groups. Each group is scaled independently based on the maximum value within that group, preserving precision for values in groups that do not contain large outliers.

Compare shared tensor scaling with grouped scaling: partitioning an activation vector allows groups without large outliers to retain more precision. section 5.6.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains maximum-based scaling, rounding, dequantization, and outlier-induced error through a worked vector example.
Examples
Uses the vector [2.0, 3.0, 500.0] and a signed 8-bit range to quantify precision loss.
What it assumes
Assumes familiarity with tensors and numerical precision while grounding the conversion through explicit arithmetic.
What you can do after
Apply shared scaling and dequantization while diagnosing precision loss caused by tensor outliers.
How it connects
Connects tensor-wise scaling with outliers, dequantization, FP8, BF16, and fine-grained quantization.

Also explained in

Compare how 2 books explain scalar quantization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.