liveBook · Manning

Top-1 prediction agreement

As explained in Quantization and Fast Inference

Top-1 prediction agreement is the fraction of inputs for which two models select the same highest-scoring class.

Top-1 prediction agreement is the fraction of inputs for which two models choose the same highest-scoring class. Compare predictions on identical inputs alongside raw-logit metrics: agreement shows whether differences change the selected class, not how much the logits differ. section 6.5

Listing 6.8 Tensor-level equivalence metrics
def compute_equivalence_metrics(baseline, variant, labels=None):
    diff = baseline - variant                                     #A
    abs_diff = np.abs(diff)
 
    # Per-sample cosine similarity, then average
    dot = np.sum(baseline * variant, axis=1)
    norm_b = np.linalg.norm(baseline, axis=1)
    norm_v = np.linalg.norm(variant, axis=1)
    cos_sim = dot / (norm_b * norm_v + 1e-12)                    #B
 
    preds_base = np.argmax(baseline, axis=1)
    preds_var = np.argmax(variant, axis=1)
    agreement = np.mean(preds_base == preds_var)                  #C
 
    return {
        "max_abs_diff": float(abs_diff.max()),
        "mean_abs_diff": float(abs_diff.mean()),
        "cosine_sim": float(np.mean(cos_sim)),
        "top1_agreement": float(agreement),
    }

The calculation takes an argmax across classes for each model, compares the resulting predictions, and averages the matches. section 6.5

Quantized model comparison

Across 100 ImageNette samples, all listed ResNet-18 variants had 100% top-1 agreement with the PyTorch FP32 baseline. BERT-base ORT Dynamic INT8 agreed on 99% of 100 SST-2 samples. section 6.5

How Quantization and Fast Inference covers it

How deep it goes
Defines the fraction and computes it by comparing argmax predictions, without measuring the magnitude of logit differences.
Examples
Reports 100% agreement across ResNet-18 variants and 99% for BERT-base ORT Dynamic INT8.
What it assumes
Requires paired model outputs for identical inputs so their highest-scoring classes can be compared.
What you can do after
Evaluate quantized variants by checking how often their predicted classes match a baseline.
How it connects
Connects argmax-based class selection with logits, quantization, and cosine similarity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.