liveBook · Manning

Baseline

As explained in AI Model Evaluation

A baseline is an existing model, method, heuristic, or reference result against which a new approach is compared.

A baseline is an existing model, method, heuristic, or reference result used to give a new approach’s measurements practical meaning. Without one, a metric such as recall@10 does not show whether performance is good, improved, or harmful to the product experience. section 3.6.3 section 2.2.2

Choose a comparison point that matches the decision you need to make:

  • Compare with the current production model or a previous version to measure practical progress. section 3.6.3
  • Use a simple heuristic, such as popularity-based recommendations, to establish a transparent minimum reference. section 3.6.3
  • Compare user segments, content types, or aggregate performance to expose regressions that an overall score can hide. section 3.6.3 section 6.2
  • Keep latency, cost, safety, fairness, calibration, and other guardrails within acceptable ranges; a higher primary metric alone is not enough. section 6.2

Movie recommendations

Suppose a new recommender achieves 12% recall@10 for cold-start users. Compare it with returning users at 28%, the previous model’s long-tail exposure of 12%, or a popularity baseline. These comparisons reveal whether the issue is a segment gap, a diversity trade-off, or a broader model regression. section 3.6.3 section 3.3.2

A baseline also anchors counterfactual evaluation: estimate how a candidate policy would perform from logged decisions, then compare its value with the baseline policy. Trust the comparison only when logging is accurate, policies overlap, and decision context is sufficiently captured. section 5.2.1

How AI Model Evaluation covers it

How deep it goes
Defines baseline choices and comparison uses, then stops short of treating offline improvement as proof of online success.
Examples
Uses production models, previous versions, popularity recommendations, user segments, long-tail exposure, and diversity thresholds as comparison points.
What it assumes
Assumes an evaluation objective, measurable metric, relevant data, and a defined decision rule for acceptable improvement or degradation.
What you can do after
Evaluate/understand whether a new model or policy improves meaningfully while preserving product, safety, fairness, and system constraints.
How it connects
Connects offline evaluation, diagnostic metrics, online evaluation, A/B testing, diversity metrics, and counterfactual evaluation.

Also explained in

Compare how 27 books explain baseline →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.