liveBook · Manning

Perturbation test

As explained in Quantization and Fast Inference

Perturbation analysis evaluates how changing or disturbing one component affects a system's output or measured behavior.

Use perturbation testing to find layers where quantization causes disproportionate accuracy loss. Quantize one layer at a time, evaluate the model, and rank layers by impact. section 3.1

Measure layer impact

Record baseline accuracy, copy the model for each layer, quantize that layer’s weights using max-absolute scaling divided by 127, then calculate baseline accuracy minus quantized accuracy. Larger drops indicate greater sensitivity. section 3.1

Common sensitivity patterns include:

  • First and last layers are often more sensitive than middle layers. section 3.1
  • Attention layers tend to be more sensitive than feedforward layers; small weight ranges can make quantization noise relatively larger. section 3.1

How Quantization and Fast Inference covers it

How deep it goes
Defines one-layer-at-a-time testing and automates comparisons against baseline accuracy, without detailing evaluation setup.
Examples
Quantizes selected layer weights with max-absolute scaling divided by 127 and measures the resulting accuracy drop.
What it assumes
Assumes a model and an accuracy-evaluation function; does not teach how to establish the evaluation setup.
What you can do after
Diagnose/debug with it by ranking layers according to accuracy loss after one-at-a-time weight quantization.
How it connects
Connects quantization sensitivity to model accuracy, layer-wise quantization, attention layers, and weight ranges.

Also explained in

Compare how 2 books explain perturbation test →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.