liveBook · Manning

Clipped straight-through estimator

As explained in Quantization and Fast Inference

A clipped straight-through estimator passes gradients for values within a specified range and sets gradients outside that range to zero.

A clipped straight-through estimator (STE) approximates gradients through fake quantization: it passes gradients for values within the representable range and zeros them for clipped values. section 5.2

The backward pass treats the quantization boundary differently from the basic STE:

  • Inside the range, pass the gradient through unchanged. section 5.2
  • Outside the range, zero the gradient so optimization does not keep pushing saturated values farther out. section 5.2

Use the clipped variant by default for fake-quantized training. Basic STE passes gradients everywhere; scaled and tanh variants are also described, but have no measured advantage on the stated workloads. section 5.2.1

Figure 5.8 Visualizing the clipped STE gradient function. Top: the fake quantization staircase with "quantizable range" and "clipped range." Bottom: gradient behavior - basic STE (dashed) passes gradient 1 everywhere; clipped STE zeros gradients in the clipped regions.

The plot shows the fake-quantization staircase alongside the clipped STE's gradient: unchanged in range and zero in clipped regions. section 5.2

How Quantization and Fast Inference covers it

How deep it goes
Defines the gradient rule and implements custom forward and backward behavior, without comparing measured performance across STE variants.
Examples
Verifies gradient flow on a tensor and contrasts basic and clipped STE behavior at representable-range boundaries.
What you can do after
Apply it by implementing a custom PyTorch autograd function that quantizes in the forward pass and gates gradients in the backward pass.
How it connects
Links fake quantization and quantization-aware training through the straight-through estimator.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.