Combined evaluation integrates performance measures, diagnostics, and simulation-based tests to judge whether a system is good enough across the product dimensions that matter. It avoids relying on one metric when user behavior, seasonality, incidents, and production complexity can expose different failure modes. section 2.1 section 2.1.4
Build the evaluation around complementary questions:
- Measure performance with task or ranking metrics such as Precision@K, NDCG, groundedness, relevance, task completion, or tool-call correctness. section 2.1
- Diagnose weaknesses by slicing results across user segments, content categories, or failure types to reveal coverage, fairness, sensitivity, or tool-use gaps. section 2.1 section 2.1.4
- Probe unobserved conditions with replay, counterfactual analysis, scripted scenarios, or simulated user tasks. section 2.1 section 2.1.4
- Track competing objectives together, then compare offline improvements with online experiments rather than treating any single score as sufficient. section 2.1.4
Streaming recommendation model
Evaluate ranking with Precision@K and NDCG, slice results by user segment and content category, and stress-test cold-start users or sudden growth in new content. Use counterfactual evaluation to estimate how another ranking policy might have changed historical engagement. Review precision alongside diversity so an apparent ranking improvement does not hide a narrower content mix. section 2.1.4
Figure 2.1 visualizes how offline evaluations serve as the central concept, with three key categories—performance evaluations, diagnostic evaluations, and simulation-based evaluations—contributing independently. These categories feed into combined evaluations, which integrate insights from all three to provide a holistic view of model behavior.
Use the evaluation anatomy to see performance, diagnostic, and simulation-based assessments feeding into one combined view of model behavior. section 2.1

