A simulation-based evaluation tests how a model or system might behave in conditions that are rare, missing from historical data, or too risky or expensive to test in production. Instead of asking only how the model performed on past data, it asks what might happen under conditions that have not been directly observed. section 2.1.3
Historical data records the decisions and user behavior that actually occurred. It cannot directly show what would have happened under a different decision or a sudden change in conditions. Simulation extends offline evaluation beyond those observations, but its results still depend on the scenarios and data used. section 2.1.3 section 2.5
Choose a simulation to match the question:
- Replay logged interactions to compare a new ranking with historical engagement signals. For an alternative policy, counterfactual or off-policy evaluation estimates how different decisions might have performed; suitable logging and propensity information are important. section 2.1.3 section 2.5
- Stress-test rare or challenging conditions, such as cold-start users, sparse histories, missing features, catalog changes, or traffic spikes. section 2.1.3
- Run LLM-powered features against scripted or synthetic prompts representing edge cases, adversarial inputs, or underrepresented scenarios. section 2.1.3
- Test agents in controlled task flows, including multi-turn conversations, tool availability failures, and unexpected inputs. section 2.1.3
Table 2.3 Common Simulation Evaluations
|
Type of Simulation Evaluation
|
Purpose
|
Example
|
|
Logged Replay Evaluations
|
Replay historical events to evaluate how a new model or system would behave based on past interactions.
|
Replay past search queries or recommendation requests and compare the new ranking output against historical engagement signals.
|
|
Counterfactual / Off-Policy Evaluations
|
Estimate how an alternative action, recommendation, ranking policy, or decision might have performed using logged data.
|
Estimate how promoting item A instead of item B might have affected clicks or purchases, assuming the right logging and propensity information are available.
|
|
Scenario-Based Stress Tests
|
Test model behavior under rare, extreme, or intentionally challenging conditions.
|
Evaluate a recommender on cold-start users, sparse histories, sudden catalog changes, missing features, or traffic spikes.
|
|
Synthetic Prompt Evaluations
|
Test LLM-powered features using scripted or synthetically generated prompts that represent edge cases or underrepresented scenarios.
|
Evaluate whether a recommendation explanation model stays grounded when given sparse watch history, ambiguous preferences, or adversarial prompts.
|
|
Agent Environment Simulations
|
Evaluate how an agent behaves when it must plan, call tools, handle errors, or complete a workflow in a controlled environment.
|
Test whether a support agent calls the right ticket lookup tool, handles a failed API response, and still drafts an accurate customer response.
|
|
Simulated User Evaluations
|
Simulate user interactions to test how a system behaves across multi-turn flows.
|
Evaluate whether an LLM-powered movie assistant can handle follow-up requests like “nothing scary,” “under 90 minutes,” and “something my kids can watch.”
|
Compare common simulation types, their purposes, and example scenarios. section 2.1.3
Changing a recommendation policy
Suppose a team wants to estimate whether promoting item A instead of item B might affect clicks or purchases. A counterfactual evaluation uses logged data to estimate the alternative outcome, assuming the right logging and propensity information are available. Missing randomization keys can introduce bias into logging-based replays. section 2.1.3 section 2.5
Use simulations alongside other evaluations rather than treating them as proof of production quality. Static offline evaluations can miss feedback loops: recommendations influence user engagement, which can change future data. When those long-term dynamics matter, supplement static evaluation with simulation-based approaches or online tests designed to measure them. section 1.8.1 section 2.1
