liveBook · Manning

Gradient

As explained in Quantization and Fast Inference

A gradient is a vector of derivatives describing how a function changes with respect to its inputs or parameters.

A gradient carries the learning signal used to update model weights. Quantization disrupts that signal: rounding has derivative zero almost everywhere, so true quantized values can leave weights with no gradient and no update. section 5.2

Quantization-aware training preserves gradients while simulating quantized values:

  • Fake quantization rounds and rescales values in the forward pass while retaining full-precision weights for gradient computation. section 5.2
  • The straight-through estimator (STE) approximates the rounding derivative as 1, passing gradients backward; a clipped STE instead zeros gradients for values outside the quantizable range. section 5.2
  • Per-channel scales can reduce clipping compared with one scale per tensor, helping more gradients survive. section 5.4.1
dqdx ≈ 1

The basic STE treats the rounding operation's derivative as approximately 1, allowing gradients to pass through. section 5.2

INT4 gradient survival in ResNet-18 conv1

With per-tensor quantization, an average of 67.4% of gradients are clipped per channel, and five channels lose all gradients. Per-channel quantization reduces the average to 31.5%, a 2.10× improvement in gradient survival. section 5.4.1

How Quantization and Fast Inference covers it

How deep it goes
Explains rounding derivatives, STE and clipping, then compares gradient survival under per-tensor and per-channel quantization.
Examples
Uses a four-value rounding example and an INT4 gradient-survival comparison on ResNet-18 conv1.
What you can do after
Diagnose why rounding blocks QAT gradients and compare quantization granularity by its effect on gradient survival.
How it connects
Connects gradients to quantization-aware training, fake quantization, the straight-through estimator and per-channel quantization.

Also explained in

Compare how 27 books explain gradient →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.