liveBook · Manning

What you will learn in AI Model Evaluation

by Leemay Nassery · Manning Early Access Program (MEAP)

Chapter by chapter

Chapter 1: Setting the stage for offline evaluations

The chapter introduction frames model evaluation as important across varied AI products, especially before models are exposed to users.

1.1 Shifting AI enthusiasts to evaluation enthusiasts · 1.2 Model product development lifecycle · 1.3 What are AI model offline evaluations? · 1.3.1 The “offline” in offline evaluations · 1.3.2 Offline evaluations for internal tools · 1.3.3 Data in its many forms · 1.3.4 Offline metric categories · 1.3.5 Many metrics to choose from · 1.4 The two layers of offline evaluations · 1.5 AI v.s heuristics · 1.6 Influencing online controlled experiments · 1.7 Practical applications of offline evaluations · 1.7.1 Offline evaluations for production observability · 1.7.2 Online-offline correlation · 1.8 When not to use offline evaluations · 1.8.1 Feedback loop dynamics · 1.8.2 Balancing offline and online approaches · 1.8.3 UX considerations · 1.8.4 Cross domain collaboration is key · 1.9 Summary

What you learn in chapter 1 →

Chapter 2: Anatomy of an offline evaluation

The introduction explains why offline evaluations provide an early check of model behavior before a new version reaches users.

2.1 The many faces of offline evaluations · 2.1.1 Performance evaluations · 2.1.2 Diagnostic evaluations · 2.1.3 Simulation-based evaluations · 2.1.4 Combining evaluations to bridge the gap · 2.1.5 How evaluation feeds development cycles · 2.2 Anatomy of an offline evaluation · 2.2.1 Data as input · 2.2.2 Designing the offline evaluation · 2.2.3 Movie recommendation example · 2.3 Common pitfalls of offline evaluations in real world settings · 2.3.1 Lack of trust · 2.3.2 Limited guidance on acceptable outcomes · 2.3.3 Friction with production systems · 2.3.4 Evaluations that don’t generalize · 2.3.5 Overfitting historical data · 2.3.6 Difficulty in interpreting metrics · 2.3.7 Difficulty in interpreting metrics · 2.4 Engineering considerations · 2.4.1 Scaling offline evaluations across teams · 2.5 Summary

What you learn in chapter 2 →

Chapter 3: Using offline evaluations as diagnostics

This introduction explains why diagnostic offline evaluations matter: they help reveal the reasons behind model behavior that performance metrics alone may miss.

3.1 Diagnosing model behavior · 3.1.1 Evaluating bias in movie recommendations · 3.1.2 How diagnostics complement performance offline evaluations · 3.1.3 Bridging the gap between model and product level understanding · 3.2 Show me the metrics · 3.2.1 Error analysis metrics · 3.2.2 Fairness and bias evaluation metrics · 3.2.3 Robustness testing metrics · 3.3 Connecting diagnostics to product impact · 3.3.1 How product leads can use diagnostic metrics for decision making · 3.3.2 Balancing the product experience · 3.4 Practical applications of diagnostic evaluations · 3.4.1 Testing model robustness against noise · 3.4.2 Diagnosing data distribution shifts · 3.4.3 Improving cold-start cases · 3.4.4 Detecting user segment disparities · 3.4.5 Diagnosing LLM and agent failures · 3.4.6 Making you think about the product experience · 3.5 Common challenges · 3.6 What makes a good diagnostic offline evaluation? · 3.6.1 Have a goal in mind · 3.6.2 Have the right data · 3.6.3 Choose the right comparison point · 3.6.4 Make sure you can take action · 3.6.5 Repeat, Repeat, Repeat · 3.7 Engineering considerations · 3.7.1 Fine-grained, detailed, logging · 3.7.2 Tooling to improve interpretability · 3.7.3 Sensitive data · 3.8 Summary

What you learn in chapter 3 →

Chapter 4: Engineering system performance evaluations

This introduction explains why evaluating an AI feature requires assessing the production system around the model, not just the model’s offline performance.

4.1 Why latency and load time really matter · 4.1.1 Illustrating the impact on user experience and business metrics · 4.1.2 Technical constraints and scaling challenges · 4.2 Engineering system performance metrics · 4.2.1 Key load time metrics · 4.2.2 Key latency metrics · 4.2.3 LLM and agent-specific performance metrics · 4.2.4 Combining load time and latency metrics for performance evaluations · 4.3 Shadow traffic · 4.3.1 How shadow traffic works · 4.3.2 Shadow traffic versus A/B testing · 4.3.3 Benefits and challenges of shadow traffic · 4.3.4 Mimicking shadow traffic · 4.4 Latency degradation experiments · 4.4.1 4.4.1 Designing latency degradation tests · 4.4.2 Key online evaluation system performance metrics · 4.4.3 Movie recommendations example · 4.5 Engineering considerations · 4.5.1 4.5.1 Make sure to capacity plan your infrastructure resources · 4.5.2 4.5.2 You’ll never regret time spent testing under realistic product conditions · 4.5.3 4.5.3 Keep in mind these performance optimization tactics · 4.6 Summary

What you learn in chapter 4 →

Chapter 5: Counterfactual evaluations

Introduces counterfactual evaluations as estimates of how an AI system might have performed had it made different decisions.

5.1 What is causal inference · 5.2 Counterfactual evaluations · 5.2.1 Anatomy of a counterfactual evaluation · 5.2.2 The assumptions hiding underneath counterfactual evaluations · 5.2.3 Why counterfactual evaluations matter · 5.3 What is counterfactual logging? · 5.3.1 How counterfactual logging relates to causal inference · 5.3.2 How much data is “enough”? · 5.3.3 Data alone is not enough · 5.4 Estimating outcomes · 5.4.1 Policy value · 5.4.2 Off-policy evaluations · 5.4.3 Designing incremental action value for your domain · 5.5 Logging best practices · 5.5.1 Log all possible actions · 5.5.2 Record action probabilities · 5.5.3 Include contextual metadata · 5.5.4 Consistent and unbiased logs are the best types logs · 5.5.5 Practical example: search engine · 5.6 Strengths and realities of counterfactual evaluations · 5.7 Common pitfalls in counterfactual logging · 5.7.1 Incomplete data logging · 5.7.2 Biased propensity scores · 5.7.3 Poor overlap between the logged policy and new policy · 5.7.4 Ignoring contextual variables · 5.8 Policy value evaluation for movie recommendations · 5.8.1 The setup: data, design, and metrics · 5.8.2 Challenges along the way · 5.8.3 Outcomes and insights · 5.8.4 Making it actionable · 5.9 LLM and agentic systems are tricker · 5.9.1 Why the action space gets messy fast · 5.9.2 Start with discrete, loggable decision points · 5.9.3 A practical example: an agent with tool routing · 5.10 Engineering considerations · 5.10.1 Balancing system performance and logging · 5.10.2 Massive amounts of data · 5.11 Summary

What you learn in chapter 5 →

Chapter 6: Evaluating models in an A/B test

The chapter introduction explains why offline evaluations are necessary but A/B tests are needed to judge a model’s effect on real users and product outcomes.

6.1 A/B testing · 6.1.1 Why A/B test · 6.2 The ‘right time’ to A/B test a model · 6.2.1 Model maturity · 6.2.2 Timelines · 6.2.3 How offline evaluations influence A/B test readiness · 6.3 Designing an A/B test · 6.3.1 Operational setup · 6.3.2 Experimental design · 6.3.3 Hypothesis and purpose · 6.3.4 Model purpose & intent · 6.3.5 Experiment validity checks before interpreting results · 6.3.6 Movie recommendation A/B test example · 6.4 Prerequisites for running A/B tests · 6.4.1 Latency monitoring · 6.4.2 Feature drift monitoring · 6.4.3 Logging still matters · 6.5 Quirks of A/B testing models · 6.5.1 Multiple model variants increase complexity · 6.5.2 Novelty effect · 6.5.3 Model warm-up and cold start effects · 6.5.4 Variance and sensitivity to noise · 6.5.5 Offline and online mismatch · 6.6 Interpreting ambiguous or low-signal results · 6.7 Engineering considerations · 6.7.1 Infrastructure-induced latency or errors · 6.7.2 If you just don't have an A/B testing platform · 6.8 Summary

What you learn in chapter 6 →

Chapter 7: From offline evaluation to live experiment

Positions A/B testing as one part of a broader model evaluation system that connects offline evaluation with online experimentation.

7.1 Model selection playbook · 7.2 Internal beta testing before exposing to users · 7.3 Model selection scoring rubric · 7.4 Model maturity nuances in selection decisions · 7.5 Stakeholder alignment and communication · 7.6 Mapping offline insights into targeted online monitoring · 7.6.1 Translating offline evaluations into online watch points · 7.6.2 Building a metric mirror table · 7.6.3 Tracking correlation between offline and online metrics over time · 7.7 Interim checks · 7.8 Engineering considerations · 7.8.1 Realtime monitoring tied to product and business metrics · 7.8.2 Side-by-side model comparison tools · 7.8.3 Reliable infrastructure for iterative improvement · 7.9 Summary

What you learn in chapter 7 →

Chapter 8: Pitfalls of online metrics

The chapter introduction argues that trustworthy AI model evaluation requires more than online metrics alone, while previewing pitfalls discussed later.

8.1 Choosing metrics that actually matter · 8.1.1 Weak metrics waste the offline investment · 8.1.2 Contract between product strategy and AI model evaluation · 8.2 Feedback loops unique to AI models · 8.2.1 How to combat feedback loops in evaluations · 8.3 Metric gaming and model exploitation · 8.3.1 Reward hacking in AI models · 8.3.2 Reward hacking isn’t limited to models · 8.3.3 Why this risk is sharper in AI A/B tests · 8.3.4 Combating reward hacking and exploitation · 8.4 The blind spots of online metrics · 8.4.1 Fairness across segments · 8.4.2 Robustness in edge cases · 8.4.3 Qualities like trust, coherence, or novelty that resist simple measurement · 8.5 The importance of the right metrics framework for AI · 8.5.1 Building metric hierarchies · 8.5.2 Revisiting metrics as models and products evolve · 8.5.3 Aligning evaluation targets with both product and model goals · 8.6 Statistical refinements for AI online metrics · 8.6.1 Top-line averages may misrepresent specific user groups · 8.6.2 Variance reduction techniques (CUPED, regression adjustment) · 8.6.3 Pitfalls of covariate choices · 8.7 Engineering considerations · 8.7.1 Logging and attribution in ensemble or blended models · 8.7.2 Monitoring for metric drift in retraining pipelines · 8.7.3 Alert fatigue · 8.7.4 Guarding against variance inflation · 8.8 Summary

What you learn in chapter 8 →

Chapter 9: LLM-as-a-judge fundamentals

Introduces LLM-as-a-judge as a way to evaluate model outputs that require semantic judgment, while stressing clear criteria and validation rather than treating the judge as objective truth.

9.1 The case for LLM-as-a-judge · 9.2 When deterministic metrics aren't enough · 9.3 What LLM judges are good at and what they are not · 9.4 Getting the goal and data right · 9.4.1 Defining the goal · 9.4.2 Designing the data · 9.5 How to design a good LLM-as-a-judge evaluation · 9.5.1 Defining the judging task · 9.5.2 Designing the prompt · 9.5.3 Choose the right judge format: scoring, ranking, or pairwise comparison · 9.5.4 Selecting the right evaluator model · 9.6 Prompt evaluation feedback loop · 9.6.1 LLM-as-a-judge for the movie recommender model · 9.7 Common LLM judge failure modes · 9.7.1 Position bias · 9.7.2 Verbosity bias · 9.7.3 Style and self-preference bias · 9.7.4 Context insufficiency · 9.7.5 Prompt injection against the judge · 9.7.6 Overconfidence and fluent explanations · 9.8 Validating an LLM-as-a-judge evaluation · 9.9 When not to use LLM-as-a-judge · 9.10 Engineering Considerations · 9.10.1 Prompt debugging 101 · 9.10.2 Document everything, just as you would any other code base · 9.10.3 Calibrate before increasing scale · 9.10.4 Observability isn’t optional, it's a must. · 9.10.5 Engineering factors influencing LLM model selection · 9.11 Summary

What you learn in chapter 9 →

Chapter 10: LLM-as-a-judge in production

The chapter introduction frames production LLM judging as a trust problem and explains why teams should expect quiet evaluation failures.

10.1 Panels and ensembles of judges · 10.1.1 It’s OK to disagree · 10.1.2 When is a panel of judges worth it? · 10.2 Decomposed and checklist-based judging · 10.2.1 Why checklists could be better than 1–5 scoring on certain tasks · 10.2.2 Writing checklist items · 10.2.3 Checklists really make debugging easier · 10.3 Reasoning patterns: chain-of-thought, debate, and critique · 10.3.1 Explicit reasoning traces before verdict · 10.3.2 Self-critique loops · 10.3.3 Debate-style judging (argue-then-judge prompting) · 10.4 Self-consistency and stability testing · 10.4.1 Consistent doesn't mean correct and correct doesn't mean consistent · 10.4.2 Repeated Sampling · 10.4.3 Swapped-order reruns · 10.4.4 Variance as a primary metric · 10.5 Escalation, abstention, and tiered judging · 10.5.1 Tiered Judging · 10.5.2 Escalation · 10.5.3 Abstain or defer scoring · 10.6 When evaluation fails: postmortems and recovery · 10.6.1 Quiet failures · 10.6.2 Triage: Figure out what actually broke · 10.6.3 Run a post mortem / incident review · 10.6.4 Rebuilding trust in the evaluation system · 10.7 Summary

What you learn in chapter 10 →

Concepts explained in this book

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.