A regression is an unintended deterioration in model or system behavior after a change. It may appear as lower evaluation scores, failing experiment gates, slower inference, higher error rates, reduced throughput, or increased cost. Treat the result as a signal to investigate, not immediate proof that the model changed for the worse. section 6.7.1 section 10.6.2
Triage a suspected regression from the evaluation pipeline outward:
- Check the data: confirm the sampling strategy, dataset snapshot, user segment, time window, and filters did not change. section 10.6.2
- Check the judge: rerun a stored canary set and compare scores against its baseline. section 10.6.2
- Check the rubric, judging prompt, and output schema for edits that could shift evaluator behavior. section 10.6.2
- Check instability with repeated sampling and swapped-order tests before treating noisy verdicts as model failure. section 10.6.2
- Check infrastructure guardrails, including latency, throughput, utilization, cost per request, cascading failures, and downstream errors. section 6.7.1
Unexpected evaluation drop
A model’s win rate falls and an A/B-test gate fails. Before declaring a model regression, compare the evaluation data with its prior snapshot, rerun the judge canary set, inspect prompt and rubric changes, and test verdict stability. If those checks are clean, investigate the model; if they are not, fix the evaluation pipeline or judge first. section 10.6.2
Use regression checks alongside product metrics rather than relying on one score. Offline metrics can drift when user behavior, catalog mix, or product surfaces change, so repeatable evaluation suites should be combined with online tests and system-health guardrails. section 2.2.2 section 6.7.1










