A metric framework connects product strategy with model evaluation. It turns broad goals such as satisfaction, retention, trust, revenue, or discovery into measurable success criteria while preventing optimization of a narrow proxy from damaging the wider experience. section 8.1.2 section 8.5.3
Build the framework as a hierarchy with distinct responsibilities:
- Choose a primary metric that represents the model's immediate objective. section 8.1.2
- Add guardrails for user and product outcomes that the treatment must not materially harm. section 8.1.2
- Use offline diagnostics to expose weaknesses and online evaluation to test whether proxy and outcome tiers move together. section 8.5
- Define acceptable trade-offs with non-inferiority margins, requiring primary-metric improvement without unacceptable guardrail decline. section 8.1.2
Movie recommendation
A recommender promoting selected movies might optimize click-through rate, while monitoring completion rate, retention, skips, and thumbs-downs. The treatment can pass only when CTR improves and consumption remains within the agreed non-inferiority margin, such as no more than 2% below control. If the system later generates explanations with an LLM, add factuality, coherence, novelty, and hallucination diagnostics, and give satisfaction and retention greater weight because persuasive text can inflate clicks without improving recommendations. section 8.1.2 section 8.5.2
Treat the framework as a living evaluation contract. Version metrics, revisit proxy validity through historical correlations with online outcomes, maintain linked datasets, and refresh diagnostics as models, products, and risks change. section 8.5.2 section 8.5.3
