liveBook · Manning

Regression

As explained in AI Model Evaluation

A regression is an unintended deterioration in a model or system behavior after a change.

A regression is an unintended deterioration in model or system behavior after a change. It may appear as lower evaluation scores, failing experiment gates, slower inference, higher error rates, reduced throughput, or increased cost. Treat the result as a signal to investigate, not immediate proof that the model changed for the worse. section 6.7.1 section 10.6.2

Triage a suspected regression from the evaluation pipeline outward:

  • Check the data: confirm the sampling strategy, dataset snapshot, user segment, time window, and filters did not change. section 10.6.2
  • Check the judge: rerun a stored canary set and compare scores against its baseline. section 10.6.2
  • Check the rubric, judging prompt, and output schema for edits that could shift evaluator behavior. section 10.6.2
  • Check instability with repeated sampling and swapped-order tests before treating noisy verdicts as model failure. section 10.6.2
  • Check infrastructure guardrails, including latency, throughput, utilization, cost per request, cascading failures, and downstream errors. section 6.7.1

Unexpected evaluation drop

A model’s win rate falls and an A/B-test gate fails. Before declaring a model regression, compare the evaluation data with its prior snapshot, rerun the judge canary set, inspect prompt and rubric changes, and test verdict stability. If those checks are clean, investigate the model; if they are not, fix the evaluation pipeline or judge first. section 10.6.2

Use regression checks alongside product metrics rather than relying on one score. Offline metrics can drift when user behavior, catalog mix, or product surfaces change, so repeatable evaluation suites should be combined with online tests and system-health guardrails. section 2.2.2 section 6.7.1

How AI Model Evaluation covers it

How deep it goes
Traces suspected failures through data, judges, prompts, rubrics, stability checks, models, and infrastructure guardrails.
Examples
Uses falling win rates, factuality scores, failed experiment gates, changed canary scores, latency, throughput, cost, and downstream errors.
What it assumes
Assumes repeatable evaluation data, stored baselines, an evaluator, and access to model and infrastructure metrics.
What you can do after
Diagnose whether an apparent regression comes from data, evaluation machinery, model behavior, or infrastructure before escalating it.
How it connects
Connects regression with offline evaluation, A/B testing, guardrail metrics, metric drift, judge stability, and model evaluation.

Also explained in

Compare how 11 books explain regression →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.