liveBook · Manning

Anomaly

As explained in Quantization and Fast Inference

An outlier is a value that differs substantially from most values in a distribution.

Activation outliers are a small number of unusually large values among mostly small activations. In large language models, they can dominate quantization ranges: clipping them can damage model quality, while preserving them can coarsen the grid for most values. section 7.0 section 2.3

Their quantization impact motivates methods that handle outlier dimensions separately or protect sensitive channels:

  • LLM.int8() routes outlier dimensions through an FP16 path alongside a dense INT8 path. section 7.7
  • AWQ rescales channels associated with high-magnitude activations before quantizing weights. section 7.7
Listing 2.6 Visualizing quantization risk
plt.hist(activations, bins=100)
plt.axvline(-alpha, color='r')                          #A
plt.axvline(alpha, color='r')                           #A
plt.show()
 

A histogram with clipping boundaries makes it possible to inspect which values fall outside the quantization range. section 2.3

How Quantization and Fast Inference covers it

How deep it goes
Explains how outliers disrupt quantization and summarizes mitigation methods without deriving their implementations.
Examples
Contrasts activations mostly in [−1,1] with a channel near 100, and describes dimensions hundreds of times larger than neighbors.
What you can do after
Evaluate/understand why clipping or coarse scaling can harm quantized model quality and motivate outlier-aware methods.
How it connects
Connects activation outliers to quantization, clipping, LLM.int8(), and AWQ.

Also explained in

Compare how 20 books explain anomaly →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.