liveBook · Manning

Straight-through estimator

As explained in Quantization and Fast Inference

A straight-through estimator approximates gradients through a nondifferentiable operation using a surrogate derivative.

Rounding blocks ordinary backpropagation because its derivative is zero almost everywhere. A straight-through estimator (STE) substitutes an approximate derivative of 1, letting gradients pass through quantization while full-precision weights update. section 5.2

The surrogate derivative determines how saturated values affect learning:

  • Basic STE passes gradients through unchanged. section 5.2
  • Clipped STE zeros gradients for values outside the quantizable range, avoiding updates to saturated values. section 5.2.2

Rounding without an STE

With inputs [0.3, 0.7, 1.2, 1.8] and scale 0.5, quantization produces [0.5, 0.5, 1.0, 2.0]. Ordinary rounding gives every input a zero gradient, so the weights cannot learn through this operation. section 5.2

Listing 8.2 Loading Llama-2-7B in NF4 with QLoRA adapters
def weight_quant(w, eps=1e-5):
    scale = 1.0 / w.abs().mean().clamp(min=eps)
    return (w * scale).round().clamp(-1, 1) / scale             #A
 
def activation_quant(x, eps=1e-5):
    scale = 127.0 / x.abs().amax(dim=-1, keepdim=True).clamp(min=eps)
    return (x * scale).round().clamp(-128, 127) / scale         #B
 
class BitLinear(nn.Linear):
    def forward(self, x):
        x_q = x + (activation_quant(x) - x).detach()            #C
        w = self.weight
        w_q = w + (weight_quant(w) - w).detach()                 #C
        return F.linear(x_q, w_q, self.bias)
 

The subtract-detach-add pattern applies the STE in a PyTorch linear layer: use quantized values in the forward pass while routing gradients to the full-precision weights. section 8.5.2

How Quantization and Fast Inference covers it

How deep it goes
Derives the zero-gradient problem, defines basic and clipped STE behavior, and provides implementations without establishing an exact derivative.
Examples
Uses a scaled four-value tensor to demonstrate zero gradients and a PyTorch BitLinear layer to show STE in code.
What it assumes
Assumes familiarity with quantization and gradients while deriving why rounding prevents gradient-based updates.
What you can do after
Build it from scratch by implementing a PyTorch fake quantizer or quantized layer that preserves gradient flow through rounding.
How it connects
Connects quantization-aware training and fake quantization with backpropagation, per-channel quantization, and ternary quantization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.