liveBook · Manning

Evaluation

As explained in Context Engineering

Evaluation is the structured testing and measurement of an AI system's behavior and performance.

Evaluation is the structured, repeatable measurement of an AI system against explicit quality criteria. For context-aware systems, it tests whether context selection and management improve behavior, replacing informal prompt inspection with evidence that can be compared as the system evolves. section 8.2

An eval combines a task input, an execution environment, and grading logic. A simple LLM eval may compare an answer with a reference; an agent eval should also inspect the trajectory and final environment state. A task defines its inputs and success criteria, while a harness runs tasks and graders. Multiple trials help assess consistency when outputs vary. section 8.2.1

Table 8.2 Common evaluation dimensions for context-aware AI systems
Dimension Evaluation focus Typical signal

Factual accuracy

Claims match the available source, reference answer, or ground truth

Supported claims and absence of unsupported claims

Completeness

The response covers the required aspects of the task

Missing constraints, omitted subquestions, or partial answers

Citation accuracy

Citations or references support the specific claims attached to them

Correct source mapping and no citation laundering

Context relevance

Selected context matches the user query and task objective

Retrieval precision, retrieval recall, and stale-context rate

Tool efficiency

The system uses appropriate tools at a reasonable cost

Unnecessary calls, retries, latency, and token consumption

Instruction and policy adherence

Higher-priority instructions and policy constraints are followed

Policy violations, unsafe completions, or ignored system constraints

Robustness

Behavior remains stable under paraphrases, reordered context, or benign noise

Sensitivity to wording, context order, irrelevant passages, or minor format changes

Final state correctness

Persistent state matches the intended result after execution

Assertions over files, databases, tickets, or configurations

Use these dimensions to choose explicit signals for context-aware behavior, from factual accuracy and completeness to retrieval relevance and tool efficiency. section 8.2

Combine graders according to what can be checked. Code-based graders are fast and reproducible but can reject valid variations. Model-based graders handle open-ended judgments but need calibration and can vary between runs. Human graders bring expert judgment but are slower and harder to scale. section 8.2.1

Table 8.3 Comparison of grader types for AI agents across speed, cost, flexibility, and reproducibility
Grader type Advantages Disadvantages

Code-based

Fast, cheap, reproducible

Brittle, lacks nuance

Model-based

Flexible, scalable, handles open-ended tasks

Non-deterministic, requires calibration

Human

Best alignment with expert judgment

Expensive, slow, difficult to scale

Compare grader types by speed, cost, flexibility, and reproducibility. section 8.2.1

Probe a compressed debugging session

After compressing a debugging session, test whether the agent can recall the original error, changed files, current test status, and agreed next step. These targeted probes assess whether the task can continue; similarity between the summary and the original is only a partial signal. section 8.2 section 7.4.1

For coding agents, evaluate both the artifact and the path that produced it. Check the artifact with tests, static analysis, security scanners, type checking, build reproducibility, and review. Inspect context traces, tool-call logs, retrieved documents, prompt templates, and decision records to assess the workflow and context received. section 11.4.1

Eval-driven development makes these checks a proactive development driver. Define the task, expected outcome, and constraints, then build an evaluation around them. Combine deterministic checks with qualitative review and, where useful, structured LLM-as-judge scoring to detect regressions and guide controlled changes. section 11.4.1

Listing 8.4 Python example for evaluation using the LLM-as-judge pattern with Ragas
from ragas import evaluate
from ragas.metrics import answer_relevancy, context_precision,
context_recall, faithfulness

# Build RAG pipeline and an evaluation dataset (details in the repo)

metrics = [ #1
    faithfulness,
    answer_relevancy,
    context_recall,
    context_precision,
]

result = evaluate( #2
    dataset=eval_dataset,
    metrics=metrics,
    llm=llm,
    embeddings=embeddings,
    raise_exceptions=False,
)

This Ragas example scores a RAG pipeline for faithfulness, answer relevancy, context precision, and context recall. APIs vary by release; calibrate scores against deterministic checks or human-labeled examples before using them as release gates. section 8.4.3

Evaluation sets the quality bar; observability preserves runtime evidence to explain failures. Turn failures into eval cases, use traces to investigate their causes, and validate changes to context management or orchestration before release. section 8.4.5

How Context Engineering covers it

How deep it goes
Defines eval components and grader trade-offs, then applies targeted probes and a library-based retrieval evaluation.
Examples
Uses debugging-session recall probes, coding-agent artifact checks, and a RAG scoring example.
What you can do after
Apply evaluation via Ragas to score retrieval and generation, calibrating model judgments before release use.
How it connects
Connects evaluation to context compression, retrieval quality, agent trajectories, and observability.

Also explained in

Compare how 29 books explain evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.