liveBook · Manning

Saturation

As explained in Quantization and Fast Inference

Saturation is the clamping of a value outside a numerical format's representable range to a boundary value.

Saturation clamps a value that exceeds a numerical format’s representable range to its nearest boundary. In fixed-point arithmetic, this limits the stored result; in serving systems, a saturation point is where capacity limits begin to affect requests. section 2.1 section 12.3.2

The practical effects depend on what is saturating:

  • In fixed-point quantization, values outside the grid are clipped, causing overload error; widening the grid reduces clipping but increases rounding error. section 2.6
  • In model serving, freed memory can move the concurrency limit outward, but only when workload demand approaches the memory limit. section 12.3.2

Clipping in Q3.4

An 8-bit Q3.4 value has a maximum of 7.9375. Storing 10.0 clips to that boundary, as does adding two stored 5.0 values. section 2.1

Figure 12.6 KV-cache occupancy (left) and request-queue depth (right) against offered concurrency on a log-2 axis, FP16 versus AWQ. FP16 is the solid line with circle markers, AWQ the dashed line with square markers. Occupancy stays below 2% and queue depth stays at zero across the entire sweep for both formats — the system never approaches its memory limit at this model size, so the saturation knee the freed memory is meant to push outward does not appear.

The figure compares FP16 and AWQ serving: at the tested 3B workload, the queue stays empty and KV-cache occupancy remains below 2%, so no saturation knee appears. section 12.3.2

How Quantization and Fast Inference covers it

How deep it goes
Works through Q3.4 clamping and quantization overload error, then measures a serving sweep where the saturation knee is absent.
Examples
Uses Q3.4 inputs of 10.0 and 5.0 + 5.0, both clipped to 7.9375, and a 3B serving sweep with no queue knee.
What it assumes
Builds from integer storage and Q-format range, introducing clamping through a fixed-point simulation.
What you can do after
Diagnose representational clipping from fixed-point limits and recognize when serving capacity has not reached its saturation knee.
How it connects
Connects fixed-point arithmetic and quantization to overflow, rounding, and KV-cache capacity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.