liveBook · Manning

Anomaly

As explained in Build a DeepSeek Model (From Scratch)

An outlier is a numerical activation, weight, or gradient whose magnitude is unusually large relative to other values in its group.

In tensor-wise quantization, one unusually large value sets the scale for every value in the tensor, potentially erasing distinctions among smaller values. section 5.6.2

A single value can erase differences

Quantizing [2.0, 3.0, 500.0] to 8-bit integers with a shared scale produces [1, 1, 127]. The first two values become indistinguishable; dequantization gives approximately [3.94, 3.94, 500.0]. section 5.6.2

Fine-grained quantization limits an outlier’s effect by scaling smaller groups independently. A large value in one group then does not determine the scale for values in other groups. section 5.6.2 section 5.6.4

Figure 5.27 Fine-Grained Quantization for an activation vector. The vector is partitioned into smaller groups. Each group is scaled independently based on the maximum value within that group, preserving precision for values in groups that do not contain large outliers.

The activation-vector diagram shows how independent group scales protect values in groups without large outliers. section 5.6.2

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Works through a numerical quantization example and explains grouped scaling as mitigation, without detailing its implementation.
Examples
Illustrates precision loss with the vector [2.0, 3.0, 500.0] quantized to 8-bit integers.
What you can do after
Understand how a large value can degrade smaller values under shared scaling and how independent group scales mitigate the effect.
How it connects
Connects outliers to tensor-wise scaling, fine-grained quantization, and mixed-precision training.

Also explained in

Compare how 20 books explain anomaly →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.