Chapter 10: Evaluating agents
This introduction explains why agent autonomy makes rigorous, automated evaluation necessary and why defining success can be difficult.
After this chapter you will be able to
- Risks of agent autonomy. Explain why an agent’s ability to choose information, tools, actions, and stopping points makes rigorous evaluation important.
- Need for automated process evaluation. Explain why evaluating only an agent’s final output is insufficient and why the evaluation must be automated.
- Criteria for open-ended tasks. Distinguish evaluation against clear ground truth from evaluation of open-ended tasks, and explain the need for an explicit rubric.
- Observability and evaluation link. Explain how making an agent’s inputs, reasoning, tool use, and outputs visible provides data for debugging and evaluation.
- Telemetry as raw evidence. Describe telemetry as operational signals collected for monitoring and analysis, and give examples of agent telemetry and what it can help measure.
- Metrics for system health. Explain what metrics measure and why they are useful for monitoring an agent, giving an example from the section.
- Request traces spans and events. Trace how a request is represented by a trace, spans, and events, distinguishing the scope and timing of each.
- Agent instrumentation pipeline. Explain how adding instrumentation code makes important agent events emit telemetry that is collected, processed, and exported for analysis.
- OpenTelemetry portability. Explain what OpenTelemetry provides and why its vendor-neutral approach, including OTLP, makes telemetry portable across observability platforms.
- Consequences of agent actions. Explain why judging an agent's answer alone is insufficient: its tool calls can change real-world state, so evaluation must consider the consequences of its actions.
- Traces and failure patterns. Trace the evaluation-design flow from representative user requests through agent runs and execution traces to human error analysis and a failure taxonomy. Describe why reviewers focus on what went wrong, especially the first error, before assigning scores.
- Rubrics and measurable metrics. Explain how the failure taxonomy leads to criteria for good agent behavior, a shared scoring rubric, and measurable metrics.
- Limits of answer-only evaluation. Explain why a correct final answer is not enough to judge an agent: the same input can lead to different paths, and a good result may be a matter of luck rather than a reliable process.
- Three evaluation perspectives. Distinguish evaluation of the final output, individual components, and trajectory, including what each perspective checks.
- Benchmark value and limits. Explain why public benchmarks are useful for establishing an agent’s baseline and comparing systems, and why a strong benchmark score does not guarantee strong product performance.
- Custom and production data. Distinguish internally constructed product-use-case datasets from evaluation data drawn from real interactions, including how each is prepared and the limitation of production logs.
- Balanced test set composition. Describe the three types of cases in a self-built test set, what each tests, and the suggested order for adding them.
- Evidence-Based Evaluation Criteria. Explain why error analysis follows running an agent on a dataset and collecting traces, and how it helps choose evaluation criteria.
- Open Coding of Traces. Describe how evaluators use open coding to record problems found in an agent interaction.
- Failure Taxonomy Construction. Explain how open-coding notes become a failure taxonomy and how that taxonomy supports evaluation design.
- Explicit binary evaluation rubrics. Explain how a rubric makes good and bad outcomes explicit so evaluators grade cases consistently, and why binary PASS/FAIL decisions are generally more stable than graded Likert ratings.
- Aggregate evaluation pass rates. Explain how metrics turn case-level rubric judgments into trackable numbers, including the overall pass rate and individual rubric-item pass rates.
- Rubric-Based LLM Judging. Explain how an LLM judge evaluates agent responses and why it helps when agents change frequently.
- Single-output grading. Explain how a judge grades one agent response against a rubric and how an optional reference answer affects that grading.
- Pairwise output comparison. Describe how pairwise judging selects between two outputs for the same input and why it suits subjective judgments and A/B testing.
- Evaluator inputs and evidence pack. Describe the inputs given to the judge: the user query, the agent’s final answer, the rubric, the materials and metadata the agent saw, and an expected answer if available. Explain that the standardized evidence pack contains relevant source excerpts and identifying metadata.
- Evidence-bound judgment. Explain why the evaluator must judge claims only against the provided evidence pack, failing unsupported key claims and citations whose excerpts do not support the cited claims.
- Three pass-fail criteria. Distinguish the three rubric checks: whether the answer addresses the query with a conclusion, whether key claims and citations align with the evidence, and whether every explicit user requirement is met. State that each item receives PASS or FAIL, with no partial credit.
- Continuous evaluation in CI/CD. Explain why production agents need automated evaluation as part of CI/CD and what risks those checks are meant to reduce.
- Predeployment quality gate. Explain why agent changes must pass quality verification before production deployment, and distinguish manual review from an automated deployment gate.
- Premerge integration checks. Describe when premerge CI runs and how its fast checks and small evaluation set guard against regressions.
- Staging validation checks. Describe what happens after CI and merging, and why staging uses heavier validation than premerge CI.
- Evaluation as an evolving product. Explain why a fixed test set and rubric cannot maintain agent quality indefinitely, and why evaluation must remain connected to product value.
- Operational feedback improvement loop. Trace how operational data feeds repeated improvements to an agent’s evaluation dataset and criteria.
What it builds on
- Agent: LLM-directed flow (chapter 1) · The distinction between an agent’s LLM-directed flow and a developer-defined workflow.
- The think-act cycle (chapter 4) · The agent’s repeated thinking and acting during a task.
Section by section
Introduction
This introduction explains why agent autonomy makes rigorous, automated evaluation necessary and why defining success can be difficult.
- Risks of agent autonomy: Explain why an agent’s ability to choose information, tools, actions, and stopping points makes rigorous evaluation important.
Why it matters: Evaluation helps protect users when an agent can take consequential actions. - Need for automated process evaluation: Explain why evaluating only an agent’s final output is insufficient and why the evaluation must be automated.
Why it matters: A correct answer alone may conceal a faulty process that fails on the next task. - Criteria for open-ended tasks: Distinguish evaluation against clear ground truth from evaluation of open-ended tasks, and explain the need for an explicit rubric.
Why it matters: Automation cannot judge quality consistently until success has been defined.
Key terms: autonomy, rigorous evaluation, automated evaluation, intermediate reasoning, tool selection, action decisions, evaluation space, clear ground truth, open-ended tasks, explicit rubric
10.1 Observing an agent
This section explains how collecting signals from an agent’s operation makes its behavior observable, evaluable, and monitorable.
- Observability and evaluation link: Explain how making an agent’s inputs, reasoning, tool use, and outputs visible provides data for debugging and evaluation.
Why it matters: Seeing what happened inside an agent makes failures easier to investigate and performance possible to evaluate over time. - Telemetry as raw evidence: Describe telemetry as operational signals collected for monitoring and analysis, and give examples of agent telemetry and what it can help measure.
Why it matters: Evaluation needs collected evidence rather than impressions of how the agent performed. - Agent-specific monitoring needs: Distinguish agent observability from traditional infrastructure monitoring, and explain how monitoring agent signals can reveal issues that need a response.
Why it matters: A system can look healthy at the infrastructure level while its agent behavior or tool calls need attention.
Key terms: observability, evaluation, telemetry, latency metrics, traces, logs, token-use statistics, traditional monitoring, agent observability, nondeterministic behavior
10.1.1 Pillars of observability: Metrics, traces, and logs
This section explains how metrics, traces, and logs provide complementary views of an agent’s behavior.
- Metrics for system health: Explain what metrics measure and why they are useful for monitoring an agent, giving an example from the section.
Why it matters: Metrics help you spot performance or reliability problems without inspecting every request. - Request traces spans and events: Trace how a request is represented by a trace, spans, and events, distinguishing the scope and timing of each.
Why it matters: This structure makes the sequence of an agent’s actions easier to understand when investigating a request. - Selective logs for debugging: Describe what logs record, why they help debugging, and why they are used selectively alongside metrics and traces.
Why it matters: Selective logs preserve useful debugging context without overwhelming the investigation.
Key terms: metrics, telemetry, dashboards, alerting systems, trace, span, event, logs, debugging
10.1.2 Generating, collecting, and exporting telemetry: OpenTelemetry
This section explains how OpenTelemetry captures and exports agent telemetry, and how observability platforms use that data for analysis.
- Agent instrumentation pipeline: Explain how adding instrumentation code makes important agent events emit telemetry that is collected, processed, and exported for analysis.
Why it matters: This connects events inside an agent to the information developers can inspect. - OpenTelemetry portability: Explain what OpenTelemetry provides and why its vendor-neutral approach, including OTLP, makes telemetry portable across observability platforms.
Why it matters: Portable instrumentation makes it easier to change observability platforms without rewriting the code that emits telemetry. - Agent signals and analysis platforms: Describe which agent activities can be instrumented and distinguish OpenTelemetry's instrumentation role from the analysis role of observability and evaluation platforms.
Why it matters: It clarifies what agent activity becomes visible and which tools help developers inspect it.
Key terms: instrumentation, telemetry data, observability backend, OpenTelemetry (OTel), vendor neutrality, OpenTelemetry Protocol (OTLP), metrics, traces, logs, LLM requests and responses
10.2 Building datasets and establishing evaluation criteria
The section explains how to turn representative agent runs and their failures into evaluation criteria that account for both answers and real-world actions.
- Consequences of agent actions: Explain why judging an agent's answer alone is insufficient: its tool calls can change real-world state, so evaluation must consider the consequences of its actions.
Why it matters: An agent can sound right while doing something wrong in the world. - Traces and failure patterns: Trace the evaluation-design flow from representative user requests through agent runs and execution traces to human error analysis and a failure taxonomy. Describe why reviewers focus on what went wrong, especially the first error, before assigning scores.
Why it matters: Examining real runs helps identify the failures that recur or cause the most damage. - Rubrics and measurable metrics: Explain how the failure taxonomy leads to criteria for good agent behavior, a shared scoring rubric, and measurable metrics.
Why it matters: Criteria grounded in observed failures make agent quality more than a vague impression.
Key terms: agent quality, tool calls, consequences of its actions, dataset, representative user requests, execution traces, error analysis, failure taxonomy, frequency, evaluation criteria
10.2.1 What should we evaluate?
The section explains why agent evaluation must examine the final result, the components, and the path taken, using representative requests that can be run repeatedly.
- Limits of answer-only evaluation: Explain why a correct final answer is not enough to judge an agent: the same input can lead to different paths, and a good result may be a matter of luck rather than a reliable process.
Why it matters: Looking beyond the answer helps distinguish dependable behavior from a lucky run. - Three evaluation perspectives: Distinguish evaluation of the final output, individual components, and trajectory, including what each perspective checks.
Why it matters: These perspectives reveal different kinds of problems that a final-answer check alone could miss. - Representative evaluation dataset: Describe why evaluation needs a collection of representative user requests that can be run repeatedly across realistic scenarios.
Why it matters: A dataset lets evaluation go beyond a single run or request.
Key terms: nondeterministic, final result, final output, component, trajectory, evaluation dataset, representative user requests
10.2.2 Creating a dataset
Explains where agent-evaluation datasets come from and how to build a test set that covers normal, difficult, and adversarial situations.
- Benchmark value and limits: Explain why public benchmarks are useful for establishing an agent’s baseline and comparing systems, and why a strong benchmark score does not guarantee strong product performance.
Why it matters: A benchmark can show progress without proving that the agent works well for its users. - Custom and production data: Distinguish internally constructed product-use-case datasets from evaluation data drawn from real interactions, including how each is prepared and the limitation of production logs.
Why it matters: Product-focused tests can expose gaps that public benchmarks miss, while real logs may need labeling before formal evaluation. - Balanced test set composition: Describe the three types of cases in a self-built test set, what each tests, and the suggested order for adding them.
Why it matters: Different case types reveal different failures, so a test set limited to normal tasks gives an incomplete picture.
Key terms: benchmarks, baseline capability, custom datasets, production data, ground-truth labels, human annotation, representative core use cases, complex and challenging edge cases, adversarial or stress-testing inputs
10.2.3 Analyzing errors
The section shows how to turn agent traces into evidence-based evaluation criteria by noting errors, grouping them into failure categories, and revisiting those categories as the agent changes.
- Evidence-Based Evaluation Criteria: Explain why error analysis follows running an agent on a dataset and collecting traces, and how it helps choose evaluation criteria.
Why it matters: It keeps evaluation focused on failures the agent actually exhibits. - Open Coding of Traces: Describe how evaluators use open coding to record problems found in an agent interaction.
Why it matters: Open-ended notes let failures emerge from what the agent actually did. - Failure Taxonomy Construction: Explain how open-coding notes become a failure taxonomy and how that taxonomy supports evaluation design.
Why it matters: Grouping individual mistakes turns trace notes into categories that can be evaluated consistently. - Taxonomy Review Cycle: State when an initial taxonomy pass is complete and when error analysis should be repeated.
Why it matters: Revisiting the taxonomy helps it reflect the agent after significant changes.
Key terms: dataset, traces, error analysis, evaluation criteria, open coding, full trace, predefined categories, failure taxonomy, rubric items, measurable metrics
10.2.4 Designing rubrics and metrics
This section turns findings from error analysis into explicit pass/fail rubrics and numerical metrics for tracking agent evaluation.
- Explicit binary evaluation rubrics: Explain how a rubric makes good and bad outcomes explicit so evaluators grade cases consistently, and why binary PASS/FAIL decisions are generally more stable than graded Likert ratings.
Why it matters: Clear pass/fail criteria help different evaluators identify the same failures. - Three pass fail criteria: Describe what counts as PASS or FAIL for answer relevancy, citation reliability, and requirement compliance.
Why it matters: These criteria distinguish an answer that sounds plausible from one that answers, supports, and follows the request. - Aggregate evaluation pass rates: Explain how metrics turn case-level rubric judgments into trackable numbers, including the overall pass rate and individual rubric-item pass rates.
Why it matters: Aggregate rates show whether performance changes over time and which criterion needs attention.
Key terms: rubric, Likert scale, binary decision, PASS, FAIL, answer relevancy, citation reliability, requirement compliance, metric, overall pass rate
10.3 Evaluating with LLM-as-a-judge
This section explains how LLM judges grade agent responses and compares them with code-based evaluation, human review, and end-user feedback.
- Rubric-Based LLM Judging: Explain how an LLM judge evaluates agent responses and why it helps when agents change frequently.
Why it matters: Rubric-based judging makes repeated evaluation practical as an agent evolves. - Code-Based Evaluation Tradeoffs: Describe what code-based evaluators can check and where their usefulness ends.
Why it matters: Code checks are useful for clear-cut requirements but cannot replace judgments about every response quality. - Human Review and Hybrid Grading: Explain why human reviewers remain valuable and how they can be combined with LLM judges.
Why it matters: Combining methods preserves human scrutiny where it helps without requiring people to grade every case. - Production User Feedback Signals: Describe what end-user feedback reveals and why it is better suited to tracking changes than measuring absolute quality.
Why it matters: User feedback can reveal a production quality shift without being mistaken for an unbiased quality score.
Key terms: LLM-as-a-judge, evaluation prompt, rubric, judge model, code-based evaluator, strict format validation, execution-based evaluation, constraint checks, human evaluation, hybrid pipeline
10.3.1 Types of LLM-as-a-judge
Introduces two common ways to use an LLM as a judge: grading one agent response against a rubric and comparing two responses to the same input.
- Single-output grading: Explain how a judge grades one agent response against a rubric and how an optional reference answer affects that grading.
Why it matters: This distinguishes grading a response on its own merits from choosing between competing responses. - Pairwise output comparison: Describe how pairwise judging selects between two outputs for the same input and why it suits subjective judgments and A/B testing.
Why it matters: Relative comparisons offer a practical way to evaluate qualities that are difficult to capture with pass/fail rules.
Key terms: single-output grading, rubric, expected output, reference answers, pairwise evaluation, relative comparison, A/B testing, A, B, or Tie, win rates
10.3.2 Building a rubric-based evaluation system
This section builds a rubric-based prompt for judging a research agent’s final answer against the user’s request and the evidence the agent saw.
- Evaluator inputs and evidence pack: Describe the inputs given to the judge: the user query, the agent’s final answer, the rubric, the materials and metadata the agent saw, and an expected answer if available. Explain that the standardized evidence pack contains relevant source excerpts and identifying metadata.
Why it matters: The judge can only assess the answer against the request and evidence it receives. - Evidence-bound judgment: Explain why the evaluator must judge claims only against the provided evidence pack, failing unsupported key claims and citations whose excerpts do not support the cited claims.
Why it matters: This prevents the judge from overlooking unsupported claims simply because they sound credible or have citations attached. - Three pass-fail criteria: Distinguish the three rubric checks: whether the answer addresses the query with a conclusion, whether key claims and citations align with the evidence, and whether every explicit user requirement is met. State that each item receives PASS or FAIL, with no partial credit.
Why it matters: Separate checks reveal whether a failure concerns the answer’s relevance, its support, or the user’s instructions. - Structured verdict and reasons: Describe the judge’s required single-JSON-object output: individual rubric verdicts, an overall verdict, and failure reasons tied to spans of the final answer and supporting source IDs. State when the overall verdict is PASS.
Why it matters: A structured verdict makes failures traceable to the answer and the evidence used to judge it.
Key terms: query, final_answer, evidence_pack, rubric, reference, citation mismatch, key claim, Answer Relevancy, Citation Reliability, Requirement Compliance
10.4 Operations: CI/CD
This section explains why automated agent evaluation belongs in CI/CD rather than being a one-time development check.
- Continuous evaluation in CI/CD: Explain why production agents need automated evaluation as part of CI/CD and what risks those checks are meant to reduce.
Why it matters: A change that looks minor can quietly make an agent less reliable in production.
Key terms: automated evaluation, CI/CD, regressions, agent reliability
10.4.1 Evaluation-gated deployment
This section explains how to make quality verification a condition of deploying an agent, using manual review or automated CI/CD gates.
- Predeployment quality gate: Explain why agent changes must pass quality verification before production deployment, and distinguish manual review from an automated deployment gate.
Why it matters: A deployment gate prevents unverified agent changes from reaching production. - Premerge integration checks: Describe when premerge CI runs and how its fast checks and small evaluation set guard against regressions.
Why it matters: Premerge checks catch broken or regressed changes before they are merged. - Staging validation checks: Describe what happens after CI and merging, and why staging uses heavier validation than premerge CI.
Why it matters: Staging can expose integration and operational failures that quick premerge checks may miss.
Key terms: quality verification, manual evaluation, automated gates, evaluation harness, CI/CD, premerge integration, CI, PR, small test set, regressed
10.4.2 Improving the test set and evaluator: Agent quality flywheel
This section explains why agent evaluation must keep evolving after deployment and how operational failures feed an agent-quality improvement loop.
- Evaluation as an evolving product: Explain why a fixed test set and rubric cannot maintain agent quality indefinitely, and why evaluation must remain connected to product value.
Why it matters: An evaluation process developers trust can help improve the product rather than merely block releases. - Operational feedback improvement loop: Trace how operational data feeds repeated improvements to an agent’s evaluation dataset and criteria.
Why it matters: Using what deployments reveal keeps evaluation aligned with the failures agents encounter in practice.
Key terms: evaluation, test set, rubric, failure patterns, agent quality flywheel, operational data, error analysis, taxonomy, rubric/metric, evaluation dataset
10.5 Summary
The chapter summary connects observable agent behavior, failure-driven evaluation criteria, and continuous deployment checks into a process for improving agent reliability.
Concepts explained in this chapter
- rubricAn explicit framework of criteria and performance standards for judging the quality or success of an output.
- evaluation datasetA collection of evaluation examples reserved for measuring a system's performance after development or tuning.
- evaluationAgent evaluation is the systematic measurement of an agent's behavior and task performance.
- evaluatorLLM-as-a-judge is an evaluation method in which a language model scores or classifies another system's outputs according to an evaluation rubric.
- trajectoryAn execution trace is a chronological record of events and intermediate activity produced during program or agent execution.
- objective functionA metric is a quantitative indicator that aggregates evaluation judgments into a value trackable across cases or over time.
- citation reliabilityCitation reliability is the degree to which an answer's key claims are supported by the provided evidence and its citations match those claims.
- relevanceRelevance is the degree to which an answer focuses on the central aspects of a user's question without unrelated content.
- requirement complianceRequirement compliance is the degree to which an answer follows the requested format, criteria, scope, length, and language.
- datasetA dataset is a structured collection of representative examples used for analysis or evaluation.
- human-in-the-loopHuman evaluation is the direct assessment of system outputs by professional graders or domain experts.
- code-based evaluationCode-based evaluation is automated grading of an agent's response through predefined rules or executable programs.
- feedbackEnd-user feedback is information about system quality or satisfaction provided by people who interact with the deployed system.
- telemetryOperational data and signals generated by a running system and transmitted for monitoring and analysis.
- failure taxonomyA failure taxonomy is a structured set of broader categories used to organize related observations about system failures.
- evaluation promptAn evaluation prompt is instruction text that specifies how a model should answer a benchmark question and format its response.
- GAIAThe GAIA dataset is a collection of benchmark problems, metadata, and associated task files for evaluating general-purpose AI assistants.
- benchmarkA benchmark is a standardized evaluation used to compare system performance on a specified set of tasks.
- error analysisError analysis is a record field explaining why a problem-solving attempt failed.
- generationA task response is a standardized A2A message returned by an agent server with the result of processing a task request.
- validationData validation is the process of checking data against specified structural, type, and constraint requirements.
- eventAn event is a structured record that wraps an execution content item with metadata about its producer and occurrence time.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.