liveBook · Manning

Simulation-based approach

As explained in AI Model Evaluation

A simulation-based evaluation assesses how a model or system might behave in hypothetical, rare, unobserved, or risky conditions by constructing representative scenarios.

A simulation-based evaluation tests how a model or system might behave in conditions that are rare, missing from historical data, or too risky or expensive to test in production. Instead of asking only how the model performed on past data, it asks what might happen under conditions that have not been directly observed. section 2.1.3

Historical data records the decisions and user behavior that actually occurred. It cannot directly show what would have happened under a different decision or a sudden change in conditions. Simulation extends offline evaluation beyond those observations, but its results still depend on the scenarios and data used. section 2.1.3 section 2.5

Choose a simulation to match the question:

  • Replay logged interactions to compare a new ranking with historical engagement signals. For an alternative policy, counterfactual or off-policy evaluation estimates how different decisions might have performed; suitable logging and propensity information are important. section 2.1.3 section 2.5
  • Stress-test rare or challenging conditions, such as cold-start users, sparse histories, missing features, catalog changes, or traffic spikes. section 2.1.3
  • Run LLM-powered features against scripted or synthetic prompts representing edge cases, adversarial inputs, or underrepresented scenarios. section 2.1.3
  • Test agents in controlled task flows, including multi-turn conversations, tool availability failures, and unexpected inputs. section 2.1.3
Table 2.3 Common Simulation Evaluations
Type of Simulation Evaluation
Purpose
Example
Logged Replay Evaluations
Replay historical events to evaluate how a new model or system would behave based on past interactions.
Replay past search queries or recommendation requests and compare the new ranking output against historical engagement signals.
Counterfactual / Off-Policy Evaluations
Estimate how an alternative action, recommendation, ranking policy, or decision might have performed using logged data.
Estimate how promoting item A instead of item B might have affected clicks or purchases, assuming the right logging and propensity information are available.
Scenario-Based Stress Tests
Test model behavior under rare, extreme, or intentionally challenging conditions.
Evaluate a recommender on cold-start users, sparse histories, sudden catalog changes, missing features, or traffic spikes.
Synthetic Prompt Evaluations
Test LLM-powered features using scripted or synthetically generated prompts that represent edge cases or underrepresented scenarios.
Evaluate whether a recommendation explanation model stays grounded when given sparse watch history, ambiguous preferences, or adversarial prompts.
Agent Environment Simulations
Evaluate how an agent behaves when it must plan, call tools, handle errors, or complete a workflow in a controlled environment.
Test whether a support agent calls the right ticket lookup tool, handles a failed API response, and still drafts an accurate customer response.
Simulated User Evaluations
Simulate user interactions to test how a system behaves across multi-turn flows.
Evaluate whether an LLM-powered movie assistant can handle follow-up requests like “nothing scary,” “under 90 minutes,” and “something my kids can watch.”

Compare common simulation types, their purposes, and example scenarios. section 2.1.3

Changing a recommendation policy

Suppose a team wants to estimate whether promoting item A instead of item B might affect clicks or purchases. A counterfactual evaluation uses logged data to estimate the alternative outcome, assuming the right logging and propensity information are available. Missing randomization keys can introduce bias into logging-based replays. section 2.1.3 section 2.5

Use simulations alongside other evaluations rather than treating them as proof of production quality. Static offline evaluations can miss feedback loops: recommendations influence user engagement, which can change future data. When those long-term dynamics matter, supplement static evaluation with simulation-based approaches or online tests designed to measure them. section 1.8.1 section 2.1

How AI Model Evaluation covers it

How deep it goes
Defines simulation types and gives use cases, while identifying logging requirements and offline evaluation pitfalls.
Examples
Illustrates alternative recommendation policies, cold-start users, adversarial prompts, and agent tool failures.
What you can do after
Evaluate and understand model behavior by selecting hypothetical scenarios and interpreting simulated results alongside other evaluations.
How it connects
Connects simulation to offline evaluation, counterfactual evaluation, feedback loops, and online testing.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Model Evaluation. The section links point to the parts of the book it draws on.