liveBook · Manning

Metric framework

As explained in AI Model Evaluation

An organized set of evaluation metrics and rules that connects model objectives with product goals, constraints, and downstream outcomes.

A metric framework connects product strategy with model evaluation. It turns broad goals such as satisfaction, retention, trust, revenue, or discovery into measurable success criteria while preventing optimization of a narrow proxy from damaging the wider experience. section 8.1.2 section 8.5.3

Build the framework as a hierarchy with distinct responsibilities:

  • Choose a primary metric that represents the model's immediate objective. section 8.1.2
  • Add guardrails for user and product outcomes that the treatment must not materially harm. section 8.1.2
  • Use offline diagnostics to expose weaknesses and online evaluation to test whether proxy and outcome tiers move together. section 8.5
  • Define acceptable trade-offs with non-inferiority margins, requiring primary-metric improvement without unacceptable guardrail decline. section 8.1.2

Movie recommendation

A recommender promoting selected movies might optimize click-through rate, while monitoring completion rate, retention, skips, and thumbs-downs. The treatment can pass only when CTR improves and consumption remains within the agreed non-inferiority margin, such as no more than 2% below control. If the system later generates explanations with an LLM, add factuality, coherence, novelty, and hallucination diagnostics, and give satisfaction and retention greater weight because persuasive text can inflate clicks without improving recommendations. section 8.1.2 section 8.5.2

Treat the framework as a living evaluation contract. Version metrics, revisit proxy validity through historical correlations with online outcomes, maintain linked datasets, and refresh diagnostics as models, products, and risks change. section 8.5.2 section 8.5.3

How AI Model Evaluation covers it

How deep it goes
Defines a metric hierarchy, trade-off margins, validation process, and revision cycle without detailing statistical implementation.
Examples
Uses movie recommendation metrics including CTR, completion rate, retention, skips, thumbs-downs, factuality, coherence, novelty, and hallucination rate.
What it assumes
Assumes familiarity with product goals, model evaluation, offline metrics, online outcomes, and proxy relationships.
What you can do after
Evaluate models with aligned success criteria, explicit guardrails, trade-off margins, and metrics that evolve with product risks.
How it connects
Ties together product strategy, model evaluation, primary metrics, guardrails, proxies, diagnostics, offline evaluation, and online evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.