An agent can give a correct answer while making several wrong tool calls. Evaluation checks both the answer and the behavior that produced it against defined criteria, making failures visible and giving teams a basis for improvement. Evaluation alone cannot fix a poorly architected agent. section 7.2 section 7.0
A test set can specify questions, expected answers, and known-bad answers. For less exact responses, a rubric can describe criteria such as factual accuracy, grounding in retrieved context, tone, and structure. Benchmark tests, grounding checks, human feedback, and evaluator agents answer different questions about agent behavior. section 7.2 section 7.0
A useful evaluation looks beyond the final response:
- Check which tools the agent selected and whether its response stayed grounded in retrieved context. section 7.2
- Inspect reasoning steps, the number of steps, latency, and cost. section 7.2
- Test how the agent behaves when something fails, as well as whether the final answer meets the rubric. section 7.2
Start with a small benchmark
A retrieval-augmented generation (RAG) agent can be tested with questions about a small knowledge collection. For example, a benchmark asks what a fictional engine spins: the expected answer contains “antimatter,” while “fusion” is a known-bad answer. An initial test uses string matching against the expected answer. This is a starting point, not a complete check of the agent’s tool use or grounding. section 7.2.2 section 7.2
In test-driven agent development, define goals and benchmarks first, then test, adjust the prompt or tools, refactor, and repeat. Expect initial failures. Because language-model outputs vary, run tests multiple times to distinguish consistent behavior from a one-off result; a practical starting rule is to run each passing benchmark three times. Make the smallest change that addresses a failure before adding more instructions or tools. section 7.2 section 7.2.2
Figure 7.2 The human process of developing agents using TDAD: first, we think of goals and benchmarks; then we test, develop the prompt, refactor, and repeat the process.
The diagram shows the repeated cycle from goals and benchmarks through testing, prompting, and refactoring. section 7.2
For ongoing use, maintain curated test sets and automated checks, including LLM-as-judge checks, and run them when prompts or models change. Trace prompts, tool calls, token counts, latency, and outcomes; collect human feedback to guide further improvements. A minimal setup can start with basic logging, tests, and guardrails. section 11.1.5
Use offline tests as an initial release gate, followed by shadow traffic, a small canary, and a full rollout with automatic rollback if service-level objectives degrade. Version prompts and tools, and record the model and tool endpoints used for each turn so incidents can be investigated more reliably. section 8.3.5














