liveBook · Manning

Per-class accuracy analysis

As explained in Quantization and Fast Inference

Per-class accuracy analysis evaluates a classifier's accuracy separately for each target class.

Measure classifier accuracy separately for each target class. Class-level results can expose failures that overall accuracy hides, including whether quantization damage is diffuse or concentrated in a class. section 5.1.2

Quantized-model results illustrate two distinct failure patterns:

  • ResNet-18 at INT4 loses accuracy across all ten ImageNette classes: mean degradation is 12.2 percentage points, with a 6.0-point standard deviation. Three classes lose over 20 points, but none collapses entirely. section 5.1.2
  • BERT at INT4 predicts positive sentiment for every SST-2 input: negative-class accuracy falls from 91.4% to 0%, while positive-class accuracy reaches 100%. Overall accuracy of 50.9% hides this collapse. section 5.1.2

How Quantization and Fast Inference covers it

How deep it goes
Compares per-class accuracy with FP32 baselines and interprets diffuse degradation versus complete class collapse.
Examples
Uses INT4 ResNet-18 on ImageNette and BERT on SST-2 to contrast diffuse degradation with class collapse.
What it assumes
Assumes class-labeled evaluation results and FP32 baseline accuracies for comparison.
What you can do after
Evaluate quantization failures by class and distinguish diffuse degradation from class collapse.
How it connects
Connects classification accuracy and post-training quantization failures to quantization-aware training decisions.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.