liveBook · Manning

Accuracy tolerance

As explained in Quantization and Fast Inference

Accuracy tolerance is the amount of predictive accuracy degradation acceptable for a model or task.

Accuracy tolerance is the predictive-accuracy degradation an application can accept. In post-training quantization (PTQ), it helps determine whether a quantized model is good enough to deploy or needs quantization-aware training (QAT). section 4.1

Use tolerance alongside the model and deployment constraints:

  • Tasks with clear classification boundaries can tolerate more quantization error; generation is more sensitive because small logit shifts compound across tokens. section 3.6
  • A large CNN targeting INT8 with a 2% tolerance is described as a PTQ-trivial case; low tolerance makes aggressive quantization riskier. section 4.1

Low tolerance raises the risk

A small, attention-heavy model targeting INT4 for medical imaging combines unfavorable architecture, size, bit-width, and tolerance factors; the example indicates it will almost certainly need QAT. section 4.1

Figure 4.2 PTQ decision framework flowchart. Starting from the target bit-width at the top, the reader follows branching decisions through model architecture checks, calibration method selection, and accuracy validation gates. Each terminal node recommends either shipping the PTQ result or escalating to QAT.

Follow the PTQ decision framework through architecture and calibration choices to accuracy validation, then decide whether to ship PTQ or escalate to QAT. section 4.1

How Quantization and Fast Inference covers it

How deep it goes
Defines acceptable accuracy degradation and compares task and deployment examples without prescribing a universal threshold.
Examples
Contrasts a large INT8 CNN with 2% tolerance against low-tolerance medical imaging using a small attention-heavy INT4 model.
What it assumes
Assumes a target model and quantization bit-width, and considers the application's acceptable accuracy degradation.
What you can do after
Evaluate whether PTQ meets an application's accuracy tolerance or whether to escalate to QAT.
How it connects
Links quantization tolerance to post-training quantization, quantization-aware training, model architecture, and bit-width.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.