liveBook · Manning

Mathematical equivalence

As explained in Quantization and Fast Inference

Equivalence is the property of two models producing the same outputs for identical inputs.

Equivalence checks whether a ported model produces the same outputs as its baseline for identical inputs. Matching accuracy alone does not establish equivalence: two models can achieve the same accuracy while disagreeing on individual predictions. section 6.5

Compare raw output tensors to measure different kinds of divergence:

  • Maximum and mean absolute differences measure worst-case and typical logit error; MSE emphasizes large errors. section 6.5
  • Cosine similarity measures output-vector direction, while top-1 agreement measures matching predictions. section 6.5
  • Use the comparisons to distinguish export error, quantization error, and incomplete operator quantization coverage. section 6.5

Same accuracy, different outputs

Across 100 ImageNette samples, ResNet-18 variants all retained 96% accuracy and 100% top-1 agreement with the baseline, despite differing in logit error. BERT-base with ORT Dynamic INT8 had 99% top-1 agreement and cosine similarity of 0.973647. section 6.5

Listing 6.8 Tensor-level equivalence metrics
def compute_equivalence_metrics(baseline, variant, labels=None):
    diff = baseline - variant                                     #A
    abs_diff = np.abs(diff)
 
    # Per-sample cosine similarity, then average
    dot = np.sum(baseline * variant, axis=1)
    norm_b = np.linalg.norm(baseline, axis=1)
    norm_v = np.linalg.norm(variant, axis=1)
    cos_sim = dot / (norm_b * norm_v + 1e-12)                    #B
 
    preds_base = np.argmax(baseline, axis=1)
    preds_var = np.argmax(variant, axis=1)
    agreement = np.mean(preds_base == preds_var)                  #C
 
    return {
        "max_abs_diff": float(abs_diff.max()),
        "mean_abs_diff": float(abs_diff.mean()),
        "cosine_sim": float(np.mean(cos_sim)),
        "top1_agreement": float(agreement),
    }

See a function for computing tensor-level equivalence metrics from baseline and variant outputs. section 6.5

How Quantization and Fast Inference covers it

How deep it goes
Measures raw-logit divergence with absolute error, MSE, cosine similarity, and top-1 agreement, and separates three causes of divergence.
Examples
Compares PyTorch FP32 baselines with quantized ResNet-18 and BERT-base variants on 100 validation samples.
What you can do after
Diagnose cross-framework divergence by comparing raw logits and prediction agreement across model artifacts.
How it connects
Connects model export, quantization, and operator coverage to output divergence.

Also explained in

Compare how 4 books explain mathematical equivalence →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.