Equivalence checks whether a ported model produces the same outputs as its baseline for identical inputs. Matching accuracy alone does not establish equivalence: two models can achieve the same accuracy while disagreeing on individual predictions. section 6.5
Compare raw output tensors to measure different kinds of divergence:
- Maximum and mean absolute differences measure worst-case and typical logit error; MSE emphasizes large errors. section 6.5
- Cosine similarity measures output-vector direction, while top-1 agreement measures matching predictions. section 6.5
- Use the comparisons to distinguish export error, quantization error, and incomplete operator quantization coverage. section 6.5
Same accuracy, different outputs
Across 100 ImageNette samples, ResNet-18 variants all retained 96% accuracy and 100% top-1 agreement with the baseline, despite differing in logit error. BERT-base with ORT Dynamic INT8 had 99% top-1 agreement and cosine similarity of 0.973647. section 6.5
Listing 6.8 Tensor-level equivalence metrics
def compute_equivalence_metrics(baseline, variant, labels=None):
diff = baseline - variant #A
abs_diff = np.abs(diff)
# Per-sample cosine similarity, then average
dot = np.sum(baseline * variant, axis=1)
norm_b = np.linalg.norm(baseline, axis=1)
norm_v = np.linalg.norm(variant, axis=1)
cos_sim = dot / (norm_b * norm_v + 1e-12) #B
preds_base = np.argmax(baseline, axis=1)
preds_var = np.argmax(variant, axis=1)
agreement = np.mean(preds_base == preds_var) #C
return {
"max_abs_diff": float(abs_diff.max()),
"mean_abs_diff": float(abs_diff.mean()),
"cosine_sim": float(np.mean(cos_sim)),
"top1_agreement": float(agreement),
}See a function for computing tensor-level equivalence metrics from baseline and variant outputs. section 6.5



