Top-1 prediction agreement is the fraction of inputs for which two models choose the same highest-scoring class. Compare predictions on identical inputs alongside raw-logit metrics: agreement shows whether differences change the selected class, not how much the logits differ. section 6.5
Listing 6.8 Tensor-level equivalence metrics
def compute_equivalence_metrics(baseline, variant, labels=None):
diff = baseline - variant #A
abs_diff = np.abs(diff)
# Per-sample cosine similarity, then average
dot = np.sum(baseline * variant, axis=1)
norm_b = np.linalg.norm(baseline, axis=1)
norm_v = np.linalg.norm(variant, axis=1)
cos_sim = dot / (norm_b * norm_v + 1e-12) #B
preds_base = np.argmax(baseline, axis=1)
preds_var = np.argmax(variant, axis=1)
agreement = np.mean(preds_base == preds_var) #C
return {
"max_abs_diff": float(abs_diff.max()),
"mean_abs_diff": float(abs_diff.mean()),
"cosine_sim": float(np.mean(cos_sim)),
"top1_agreement": float(agreement),
}The calculation takes an argmax across classes for each model, compares the resulting predictions, and averages the matches. section 6.5
Quantized model comparison
Across 100 ImageNette samples, all listed ResNet-18 variants had 100% top-1 agreement with the PyTorch FP32 baseline. BERT-base ORT Dynamic INT8 agreed on 99% of 100 SST-2 samples. section 6.5
