Evaluation design specifies how to assess a model or AI system against an objective. It connects input data to results by defining what to measure, which scenarios to test, how to sample, and how much confidence to place in the findings. The design should support a decision, not just produce a high metric. section 2.2 section 2.2.2
A practical offline evaluation design makes these choices explicit:
- Inputs: Select historical interactions, ground-truth labels, and contextual metadata relevant to the objective. section 2.2
- Measures and scenarios: Define metrics, evaluation granularity, and conditions such as cold-start users, ambiguous prompts, or failed tool calls. section 2.2 section 2.2.2
- Sampling and confidence: Specify how cases are sampled and confidence is estimated. section 2.2
- Decision rules: Set what improvement is meaningful, how much guardrail degradation is acceptable, and which failures block launch. section 2.2.2
- Reproducibility: Document and version datasets, models, code, metric definitions, and other relevant artifacts, such as prompts, rubrics, judge models, and evaluation configuration. section 2.2.2
Table 2.4 Offline Evaluation Input/Design/Output Summary
|
Element
|
Description
|
Example
|
|
Input
|
The data fed into the evaluation process. This forms the foundation of the offline evaluation.
|
- User interaction logs (JSON, Parquet)
- Ground truth labels (binary, multi-class)
- Metadata (device_type, location)
|
|
Design
|
The structure of the evaluation: how it's run, what it tests, and how results are calculated.
|
- Metric definitions such as Precision@10 or NDCG@5
- Simulation setup such as cold-start users or missing feature tests
- Evaluation granularity such as per-session or per-user
|
|
Output
|
The results of the evaluation: quantitative and diagnostic signals about model behavior.
|
- Performance metrics
- Diagnostic insights
- Visual reports or dashboards
|
Use the input/design/output summary to connect concrete data formats, design choices, and diagnostic results. section 2.2
Movie recommendations
For a recommendation model, use interaction logs, labels, and user metadata as inputs. Define Precision@10 and test cold-start users, specifying whether results are calculated per user or per session. Interpret the resulting performance and diagnostic signals against the product objective before deciding whether to iterate or proceed to online testing. section 2.2 section 2.2.2
Offline results can mislead if evaluation data leaks into training, prompts, features, or labels, or if historical metrics do not reflect the product goal. Use careful splits, time-aware design, and versioned artifacts. An online A/B test compares a treatment with a control to measure what happens with real users; if results disagree with offline findings, inspect segments, logs, inputs, exposure, and the offline design before settling on a cause. section 2.5 section 6.5.5 section 6.1
When comparing multiple online variants, give each a clear hypothesis and plan for interpreting results. More variants require more traffic or time and increase the risk of false positives; corrections such as Bonferroni or false discovery rate can account for multiple comparisons. section 6.5.1
