Chapter 7: Building robust agents with evaluation and feedback

Introduction

The introduction explains why evaluation and feedback matter for robust agents, what kinds of signals they provide, and why they should be considered throughout development.

• Evaluation and robustness limits: Explain how evaluation and feedback make agent behavior visible and robustness measurable and improvable, while not fixing flaws in an agent’s architecture.
  Why it matters: Evaluation helps you see what to improve without being a substitute for sound architecture.
• Varied evaluation and feedback sources: Describe the range of evaluation and feedback sources named here and explain why a single source does not answer every question about an agent.
  Why it matters: Different signals reveal different aspects of an agent’s behavior.
• Early evaluation and feedback: Justify considering evaluation and feedback when hardening an agent, rather than waiting until it is a production requirement.
  Why it matters: Treating evaluation as an ongoing part of hardening gives you opportunities to improve before production.

Key terms: evaluation, feedback, robustness, benchmark, red team testing, grounding checks, evaluator agents, human reviewers, test outputs, evaluation and feedback

7.1 Introducing agent evaluation and feedback

This section maps how agents receive evaluation and feedback, matches evaluation methods to different kinds of correctness, and explains why multi-agent evaluation needs governance.

• Feedback paths around an agent: Explain how evaluation can influence an agent immediately or be stored for later analysis.
  Why it matters: The destination of feedback determines whether it changes the current workflow or supports later improvement.
• Evaluation forms and testing purposes: Distinguish exact-match tests, LLM or agent judgments, and human judgments, and describe what red team and benchmark tests assess.
  Why it matters: Choosing an evaluation method that fits the question makes its results more useful.
• Production human feedback: Explain why user feedback matters in production and why raw ratings or comments need processing before they guide decisions.
  Why it matters: User reactions are valuable evidence, but individual reactions alone can mislead.
• Grounding and critic agent roles: Distinguish grounding an output against an authoritative source from a critic agent’s broader evaluation of reasoning, goals, and actions.
  Why it matters: These evaluators answer different questions about an agent’s work.
• Collusion risk and independent checks: Explain agent collusion and describe how evaluator diversity and human review reduce its risk.
  Why it matters: An evaluator that shares the agent’s mistakes can create false confidence instead of catching errors.
• Multi-agent evaluation governance: Describe the governance rules that keep disagreements and failures in a multi-agent evaluation system accountable.
  Why it matters: Evaluation needs accountable decisions, especially when evaluators disagree or make mistakes.

Key terms: Learn step, in-loop evaluators, evaluation database, deterministic test cases, LLM and agent evaluators, human evaluators, red team testing, benchmark testing, human feedback, statistical aggregation

7.2 Implementing test-driven agent development

This section explains how to define benchmarks before building an agent, test both its answers and behavior, and iteratively improve it without losing earlier successes.

• Upfront agent success criteria: Explain how test-driven agent development defines good responses before building the agent, using benchmarks with expected and known-bad responses and rubrics that state response criteria.
  Why it matters: Agreeing on what counts as good before changing the agent gives development a target.
• Whole-run agent evaluation: Describe why agent tests examine intermediate behavior as well as final answers, including tool choices, grounding, steps, cost, latency, and failures.
  Why it matters: Checking the whole run can expose problems that an answer-only test would miss.
• Repeated tests and minimal fixes: Trace the response to an initial benchmark failure: rerun the test to check consistency, then try the smallest prompt or tool change likely to fix it, escalating only if needed.
  Why it matters: Confirming failures and making restrained fixes avoids reacting to one-off results with bloated prompts.
• Regression testing and tradeoffs: Explain why previously passing benchmarks must be rerun after changes and how conflicting benchmarks or an impractical 100% pass rate can affect the target.
  Why it matters: A fix is only useful if its effects on existing requirements are checked.
• Specialization versus single-agent refactoring: Distinguish when a failing subset might justify a specialist agent from when refactoring the existing agent is preferable, including the new routing and evaluation costs of specialization.
  Why it matters: Adding an agent can create more failure points than it removes.
• Agent and system benchmarks: Describe why TDAD should test both specialized agents and the broader agent system at different levels of granularity.
  Why it matters: Tests at both levels can catch gaps before the system is used in production.

Key terms: test-driven agent development, TDAD, rubrics, input benchmarks, expected responses, known-bad responses, intermediate behavior, tool calls, grounded, latency

7.2.1 Exploring TDAD in practice

The section illustrates test-driven agent development with a RAG agent that answers benchmark questions and has its answers evaluated.

• RAG question-answering path: Trace how the example RAG agent turns a question into a knowledge-base search and uses the result to answer.
  Why it matters: This is the behavior the example’s benchmarks are meant to check.
• Benchmark answer evaluation: Explain how the example benchmark and evaluator determine whether the RAG agent passed or failed.
  Why it matters: It connects the agent’s answer to a checkable benchmark result.

Key terms: RAG agent, searchable words and phrases, knowledge base, context, benchmarks, evaluator, expected and incorrect answer candidates, passed or failed

7.2.2 Coding and testing the RAG agent

This section establishes a minimal benchmark-tested RAG agent and explains why apparent passes must be repeated to check consistency.

• Minimal benchmark testing setup: Describe how the initial agent is run through benchmark questions and its answers checked against expected answers. Explain why the agent is deliberately kept minimal at this stage.
  Why it matters: A minimal starting point makes benchmark results useful for guiding what the agent needs next.
• Repeated passes for consistency: Explain why a passing agent benchmark should be run again and how output variability affects the number of runs.
  Why it matters: Repeating a pass helps distinguish reliable performance from a lucky answer.

Key terms: benchmarks table, string matching, role/persona, TDAD, benchmarks, TDD, LLMs, variability, consistency

7.2.3 Refactoring the agent

This section uses benchmark results and traces to refactor a RAG agent, then shows how an overly strict evaluator can make correct answers appear wrong.

• Prompt and metadata responsibilities: Explain why the agent prompt need not repeat tool and output-format details, and where those details should normally be specified instead.
  Why it matters: Separating these responsibilities makes tools and formats easier to change without rewriting the main instructions.
• Benchmarking prompt changes: Explain why each change to an agent’s instructions should be evaluated against benchmarks before becoming the new baseline.
  Why it matters: A plausible prompt improvement is not evidence of an actual improvement until it is tested.
• Retrieval failure diagnosis: Trace why the refactored agent misses answers despite a working retrieval system, and describe the missing recovery behavior.
  Why it matters: The trace points to the agent’s response to poor results rather than a broken retrieval system.
• Keyword tool guidance: Describe how the section makes the retrieval tool easier for the agent to use and when prompt-based tool guidance is necessary.
  Why it matters: Clear tool metadata helps the agent form suitable searches without adding avoidable instructions to its main prompt.
• Evaluator false failures: Distinguish incorrect agent answers from correct answers rejected by strict equality, and explain the evaluator fix and follow-up check.
  Why it matters: A misleading score can prompt unnecessary agent changes when the evaluator is what needs fixing.

Key terms: tool metadata, output type, structured output, instruction prompt, benchmarks, baseline, agent’s instructions, trace output, retrieval, query-refinement step

7.2.4 Extending evaluation with an agent evaluator

This section replaces a word-matching benchmark evaluator with an agent evaluator that scores natural-language RAG answers, then notes a limitation of that score.

• Natural-language benchmark scoring: Explain why the RAG agent no longer has to give a single-word answer and how the agent evaluator can still use the existing benchmark tests.
  Why it matters: The agent can give useful, complete answers without requiring a new set of benchmark tests.
• Evaluator input in the loop: Trace what the evaluation loop sends to the agent evaluator after the RAG agent answers a question.
  Why it matters: The evaluator needs the answer and benchmark context together to judge the response.
• Limits of benchmark success: Describe what a favorable benchmark result can miss in the RAG agent’s answer.
  Why it matters: A passing answer can still contain unsupported additions.

Key terms: agent evaluator, strongly typed pass/fail result, key term, benchmark, evaluation loop, dictionary, expected key term, wrong answer example, benchmark score, database

7.3 Employing grounding, critic, and evaluation agents

This section distinguishes grounding, critic, and rubric-based evaluation of agent output and explains why failed evaluations need context-appropriate handling.

• Grounding of retrieval answers: Explain how a grounding agent checks a RAG or retrieval agent’s answer against the context used to produce it and what can happen when the answer is not grounded.
  Why it matters: It helps prevent a retrieval agent from presenting an answer unsupported by the material it used.
• Critic feedback and retries: Describe how a critic uses criteria or a rubric to evaluate generated content and handles content that fails.
  Why it matters: A critic can keep unacceptable content from being delivered without allowing regeneration to continue indefinitely.
• Retry ceiling failure choices: Justify why reaching a retry ceiling requires an explicit failure-handling choice, and describe how that choice varies by use case.
  Why it matters: The action taken after repeated failure determines whether the system fails safely or conceals an unacceptable result.
• Rubric feedback for complex output: Explain why rubrics are used for complex output combining retrieval and novel generation, and why evaluation may accompany the output.
  Why it matters: Feedback can help shape complex responses while giving users information to assess results that are difficult to diagnose automatically.

Key terms: grounding agent, RAG, retrieval agent, context, critics, criteria, rubric, regeneration, retry ceiling, static fallback response

7.3.1 Reviewing the grounding agent

This section reviews how a grounding agent checks a RAG answer against shared context and how its results can be handled.

• Grounding agent and shared context: Explain that grounding checks whether an answer is supported by context and citations, and describe why both the RAG agent and the grounding agent receive the retrieved context.
  Why it matters: A grounding check needs the evidence the answer was based on.
• Grounding result handling options: Describe the ways a system can use a grounding result, including feedback to the agent, approving or blocking an answer, and including the result in the final answer.
  Why it matters: Checking an answer is useful only when the system has a way to act on the result.

Key terms: grounding, retrieved knowledge context, RAG agent, grounding guardrail agent, grounding results, feedback, critic, approve or block, static answer

7.3.2 Grounding the RAG agent

This section shows how a reusable grounding agent uses the context behind a RAG answer to provide feedback, then checks that feedback against benchmark outputs.

• Context-based grounding feedback: Explain that the grounding agent provides feedback on how well an answer is grounded by retrieving the last context used to answer the question. Describe why the agent should remain generic.
  Why it matters: Grounding feedback helps reveal whether an answer relies on the context the knowledge agent was given.
• Benchmark grounding checks: Describe how the example uses benchmark questions to gauge the grounding agent: inspect the grounding output alongside the final answer to see whether the answer relies only on the provided context.
  Why it matters: Looking at both outputs for each benchmark question makes the grounding feedback assessable in this example.

Key terms: grounding agent, last context, knowledge agents, agent benchmarks, grounding output, final output, provided context

7.3.3 Implementing grounding agents as guardrails

This section shows how to use a grounding agent as an output guardrail for a RAG agent and handle answers that fail the grounding check.

• Grounding guardrail tripwire: Trace how the grounding agent checks a RAG answer and signals when it is not grounded in the available context.
  Why it matters: A failed grounding check must be detectable before an unsupported answer is returned.
• Handling ungrounded RAG answers: Describe how the main function handles a tripped grounding guardrail and distinguish the shown static response from the alternative of retrying with feedback.
  Why it matters: Catching the guardrail failure lets the application avoid delivering an invalid or potentially dangerous answer.

Key terms: grounding agent, output guardrail, tripwire-triggered property, context, try/except block, static response, feedback, RAG agent

7.3.4 Understanding the role of rubrics in evaluation

This section explains how to choose evaluation methods for different agent outputs, build and validate rubrics, and retain the evidence needed to improve evaluations.

• Output-specific evaluation methods: Distinguish how to evaluate structured outputs with known targets from open-ended natural-language outputs, and explain why a mixed agent may need both approaches.
  Why it matters: Choosing a measure that fits each output avoids unnecessary complexity and misleading results.
• Rubric design and application: Describe how to construct and apply a rubric for agent output, from setting objectives and measurable criteria through rating levels and scores.
  Why it matters: Explicit criteria and performance descriptions make judgments of open-ended output more useful and repeatable.
• Rubric validation and revision: Explain why a rubric must be checked against human judgment and revised, even if its scores appear consistent.
  Why it matters: A stable-looking score is not useful if the rubric repeatedly rewards the wrong things.
• Evaluation scores and trace logs: Explain why evaluation systems should retain scores and log both agent and evaluator activity.
  Why it matters: Scores show what needs improvement, while logs help explain what happened when an agent or its evaluator fails.

Key terms: structured outputs, known target, accuracy, precision, recall, F1 score, rubric, objectives, criteria, rating scale

7.3.5 Building a rubric critic agent

This section uses an image-generation example to show how a rubric critic evaluates outputs, controls retries, and uses a calibrated passing threshold.

• Image generation critic loop: Trace how the image-generation agent and critic work together: the generator produces an image using style guidelines, the critic evaluates its style against a rubric, and a failed image keeps the loop running.
  Why it matters: The critic turns a style guideline into a check that determines whether an image is accepted.
• Rubric criteria and useful feedback: Explain why a rubric gives more useful feedback than a bare pass/fail judgment and distinguish surface-quality criteria from behavioral criteria for agent outputs.
  Why it matters: Well-chosen criteria reveal what to improve instead of merely rejecting an output.
• Image style rubric construction: Describe how the example turns image style guidelines into an evaluation rubric by defining criteria and a described 1–5 scale, scoring the output, and asking the LLM to grade before deciding pass or fail.
  Why it matters: A defined scale and criteria make the critic’s decision more consistent and interpretable.
• Passing threshold calibration: Explain why the example accepts scores of 2 out of 5 or higher, what goes wrong when a production threshold is too low or too high, and how human-labeled outputs can help set and revalidate it.
  Why it matters: A poorly chosen cutoff can either release bad results or waste retries rejecting acceptable ones.

Key terms: ImageGenerationTool, critic agent, style guidelines rubric, describe_image, rubric, criteria, pass/fail evaluation, hallucination rate, tool usage correctness, style guidelines

7.4 Phoenix for evaluation and feedback

This section introduces Phoenix as a way to observe and evaluate agents, explains why trace-level visibility matters, and places it among other observability options.

• Trace-Level Agent Observability: Explain why maturing agent systems need more than basic dashboard traces and what Phoenix makes visible.
  Why it matters: Seeing what happened within an agent process makes difficult failures easier to investigate.
• Captured Activity and Evaluation: Describe how Phoenix relates recorded agent activity to experiments and evaluators, without assuming a particular evaluation procedure.
  Why it matters: Recorded activity can become the basis for evaluating changes, not just inspecting past runs.
• Phoenix Among Observability Options: Distinguish why the chapter uses Phoenix from the claim that it is the only viable observability platform.
  Why it matters: You can apply the chapter’s observability ideas without treating its chosen product as mandatory.

Key terms: trace-level observability, Arize Phoenix, token usage, tool usage, latency, experiments, evaluators, metadata, annotations, Phoenix

7.4.1 Connecting to Phoenix

Shows how to connect OpenAI Agents tracing to Phoenix and inspect agent activity in its dashboard.

• Phoenix tracing connection: Describe how the section connects agent tracing to Phoenix: clear the default OpenAI trace processors, add the Phoenix trace provider, and name and trace the agent or workflow.
  Why it matters: This connection makes an agent’s activity available for inspection in Phoenix.
• Phoenix trace inspection: Describe how to open the local Phoenix dashboard and what agent activity the Traces page lets you inspect.
  Why it matters: The dashboard lets you inspect what happened during an agent run rather than relying only on its final output.

Key terms: Phoenix, OpenTelemetry, set_trace_processors, Phoenix trace provider, trace function, Agents project, Traces page, agent workflow, latency, token usage

7.4.2 Adding metadata and session tracking

Shows how to attach session and metadata context to agent traces, then use Phoenix to find traces relevant to an evaluation.

• Trace metadata and session context: Explain how the section attaches metadata and session tracking to an agent workflow’s traces, including the placement of the tracing blocks.
  Why it matters: Shared trace context makes activity from an agent workflow easier to examine together.
• Phoenix metadata trace filtering: Describe how to show the Metadata column in Phoenix and use its fields to filter traces for specific agent activity.
  Why it matters: Useful metadata lets you narrow the traces you inspect to the agent activity you want to evaluate.

Key terms: using_session, using_metadata, agent trace, LLM traces, Phoenix, Agents project, Metadata column, Columns selector

7.4.3 Experimenting with evaluators

This section shows how to turn Phoenix traces into a dataset and run a basic evaluation experiment as part of TDAD’s feedback loop.

• TDAD evaluation outer loop: Explain how Phoenix experimentation fits TDAD’s outer loop: real traces become datasets, evaluators run on them, and results feed back into prompts, tools, and policies.
  Why it matters: It explains why creating an experiment is useful, not just how to click through the dashboard.
• Phoenix dataset experiment workflow: Describe how to select spans in Phoenix, add them to a new dataset, and use Run Experiment to obtain Python code for a basic task and evaluator. State what kinds of spans the section says can be evaluated.
  Why it matters: It gives you a concrete path from recorded spans to an evaluation you can run.

Key terms: Phoenix, outer loop, TDAD, traces, datasets, evaluators, spans, Add to Dataset, Create Dataset, Run Experiment

7.4.4 Providing feedback with annotations

This section shows how Phoenix annotations turn feedback on agent traces into structured signals for analysis and regression testing.

• Creating and applying Phoenix annotations: Describe how to create annotations in Phoenix and apply them to a selected trace span, typically one containing an LLM response.
  Why it matters: Attaching feedback to the relevant span preserves what the reviewer was judging.
• Structured feedback for trace analysis: Explain how reviewers' labels become queryable trace data and how filtering by annotation helps investigate errors and their relationship to operational metrics.
  Why it matters: Structured labels make recurring failures easier to find and investigate than isolated reviewer comments.
• Annotated failures as regression tests: Explain how failures flagged across sessions can become a regression test set after an agent change.
  Why it matters: Recorded failures give you concrete cases to revisit when improving an agent.
• Evaluation before production promotion: State the section's recommendation about evaluation and feedback before promoting agents into production, and why it gives that advice.
  Why it matters: The section treats feedback as a prerequisite for a robust production agent, not merely an optional dashboard feature.

Key terms: annotations, Traces page, span, LLM responses, structured labels, queryable signals, trace data, latency, cost, token usage

7.5 Exercises

This section gives five hands-on exercises that take a RAG agent from baseline benchmarking through refactoring, evaluation, grounding checks, and Phoenix tracing.

7.6 Summary

The chapter summary brings together testing, automated evaluation, grounding, and observability as an ongoing feedback loop for building robust agents.