An evaluator judges an agent’s output against benchmark criteria and records whether the result passes or fails. In a retrieval-augmented agent, it compares the returned answer with expected and incorrect candidates. section 7.2.1
Use evaluators as an engineering feedback loop:
- Pair each benchmark question with expected and incorrect answers. section 7.2.1
- Run the agent, pass its answer to the evaluator, and calculate benchmark accuracy from the results. section 7.2.1
- Inspect failures before changing the agent; an apparent retrieval failure may instead expose a brittle evaluator. section 7.2.3
- Normalize natural-language answers by lowercasing, removing trailing punctuation, and stripping unnecessary whitespace before comparing them. section 7.2.3
- Rerun a perfect benchmark to check consistency before declaring the agent reliable. section 7.2.3
Strict equality creates a false failure
An agent answers “Photons.” when the expected answer is “photons.” A strict equality check marks the response wrong, although normalization makes it correct. When phrasing can vary more substantially, an LLM-as-judge evaluator can handle multiple acceptable answers. section 7.2.3
Figure 7.3 Architecture of a simple RAG knowledge agent. The goal of the agent is to answer questions from a benchmark by searching its knowledge base and returning answers. A simple evaluator LLM evaluates the answer based on the expected and incorrect answers from the benchmark test. The accuracy of the RAG agent can be determined by the percentage of benchmark questions it answers correctly.
Trace the evaluator’s position after retrieval and answer generation in a simple RAG benchmark architecture. section 7.2.1
Figure 7.10 Viewing the dataset and clicking Run Experiment shows the generated code. You can copy the code into a Python file to create an experiment or evaluation.
Inspect generated Python for running a task and evaluator across dataset spans in an experiment. section 7.4.3
