Evaluation is the structured, repeatable measurement of an AI system against explicit quality criteria. For context-aware systems, it tests whether context selection and management improve behavior, replacing informal prompt inspection with evidence that can be compared as the system evolves. section 8.2
An eval combines a task input, an execution environment, and grading logic. A simple LLM eval may compare an answer with a reference; an agent eval should also inspect the trajectory and final environment state. A task defines its inputs and success criteria, while a harness runs tasks and graders. Multiple trials help assess consistency when outputs vary. section 8.2.1
Table 8.2 Common evaluation dimensions for context-aware AI systems
| Dimension | Evaluation focus | Typical signal |
Factual accuracy |
Claims match the available source, reference answer, or ground truth |
Supported claims and absence of unsupported claims |
Completeness |
The response covers the required aspects of the task |
Missing constraints, omitted subquestions, or partial answers |
Citation accuracy |
Citations or references support the specific claims attached to them |
Correct source mapping and no citation laundering |
Context relevance |
Selected context matches the user query and task objective |
Retrieval precision, retrieval recall, and stale-context rate |
Tool efficiency |
The system uses appropriate tools at a reasonable cost |
Unnecessary calls, retries, latency, and token consumption |
Instruction and policy adherence |
Higher-priority instructions and policy constraints are followed |
Policy violations, unsafe completions, or ignored system constraints |
Robustness |
Behavior remains stable under paraphrases, reordered context, or benign noise |
Sensitivity to wording, context order, irrelevant passages, or minor format changes |
Final state correctness |
Persistent state matches the intended result after execution |
Assertions over files, databases, tickets, or configurations |
Use these dimensions to choose explicit signals for context-aware behavior, from factual accuracy and completeness to retrieval relevance and tool efficiency. section 8.2
Combine graders according to what can be checked. Code-based graders are fast and reproducible but can reject valid variations. Model-based graders handle open-ended judgments but need calibration and can vary between runs. Human graders bring expert judgment but are slower and harder to scale. section 8.2.1
Table 8.3 Comparison of grader types for AI agents across speed, cost, flexibility, and reproducibility
| Grader type | Advantages | Disadvantages |
Code-based |
Fast, cheap, reproducible |
Brittle, lacks nuance |
Model-based |
Flexible, scalable, handles open-ended tasks |
Non-deterministic, requires calibration |
Human |
Best alignment with expert judgment |
Expensive, slow, difficult to scale |
Compare grader types by speed, cost, flexibility, and reproducibility. section 8.2.1
Probe a compressed debugging session
After compressing a debugging session, test whether the agent can recall the original error, changed files, current test status, and agreed next step. These targeted probes assess whether the task can continue; similarity between the summary and the original is only a partial signal. section 8.2 section 7.4.1
For coding agents, evaluate both the artifact and the path that produced it. Check the artifact with tests, static analysis, security scanners, type checking, build reproducibility, and review. Inspect context traces, tool-call logs, retrieved documents, prompt templates, and decision records to assess the workflow and context received. section 11.4.1
Eval-driven development makes these checks a proactive development driver. Define the task, expected outcome, and constraints, then build an evaluation around them. Combine deterministic checks with qualitative review and, where useful, structured LLM-as-judge scoring to detect regressions and guide controlled changes. section 11.4.1
Listing 8.4 Python example for evaluation using the LLM-as-judge pattern with Ragas
from ragas import evaluate
from ragas.metrics import answer_relevancy, context_precision,
context_recall, faithfulness
# Build RAG pipeline and an evaluation dataset (details in the repo)
metrics = [ #1
faithfulness,
answer_relevancy,
context_recall,
context_precision,
]
result = evaluate( #2
dataset=eval_dataset,
metrics=metrics,
llm=llm,
embeddings=embeddings,
raise_exceptions=False,
)
This Ragas example scores a RAG pipeline for faithfulness, answer relevancy, context precision, and context recall. APIs vary by release; calibrate scores against deterministic checks or human-labeled examples before using them as release gates. section 8.4.3
Evaluation sets the quality bar; observability preserves runtime evidence to explain failures. Turn failures into eval cases, use traces to investigate their causes, and validate changes to context management or orchestration before release. section 8.4.5




























