Rounding blocks ordinary backpropagation because its derivative is zero almost everywhere. A straight-through estimator (STE) substitutes an approximate derivative of 1, letting gradients pass through quantization while full-precision weights update. section 5.2
The surrogate derivative determines how saturated values affect learning:
- Basic STE passes gradients through unchanged. section 5.2
- Clipped STE zeros gradients for values outside the quantizable range, avoiding updates to saturated values. section 5.2.2
Rounding without an STE
With inputs [0.3, 0.7, 1.2, 1.8] and scale 0.5, quantization produces [0.5, 0.5, 1.0, 2.0]. Ordinary rounding gives every input a zero gradient, so the weights cannot learn through this operation. section 5.2
Listing 8.2 Loading Llama-2-7B in NF4 with QLoRA adapters
def weight_quant(w, eps=1e-5):
scale = 1.0 / w.abs().mean().clamp(min=eps)
return (w * scale).round().clamp(-1, 1) / scale #A
def activation_quant(x, eps=1e-5):
scale = 127.0 / x.abs().amax(dim=-1, keepdim=True).clamp(min=eps)
return (x * scale).round().clamp(-128, 127) / scale #B
class BitLinear(nn.Linear):
def forward(self, x):
x_q = x + (activation_quant(x) - x).detach() #C
w = self.weight
w_q = w + (weight_quant(w) - w).detach() #C
return F.linear(x_q, w_q, self.bias)
The subtract-detach-add pattern applies the STE in a PyTorch linear layer: use quantized values in the forward pass while routing gradients to the full-precision weights. section 8.5.2
