liveBook · Manning

Test suite

As explained in Context Engineering

A test suite is a collection of software tests run to check whether a program behaves as expected.

An evaluation suite is a collection of tasks designed to measure specific capabilities or behaviors of an AI agent. Each task defines inputs and success criteria; multiple trials can help estimate consistent behavior when outputs vary between runs. section 8.2.1

An evaluation harness runs the evaluations end to end. Graders score the agent’s transcript or its final environmental outcome. The transcript can include outputs, tool calls, intermediate results, and plans or reasoning artifacts that the system intentionally emits. section 8.2.1

Suite design and scoring serve different purposes:

  • A capability suite tests challenging but solvable tasks the agent does not yet perform consistently. Its failures should reveal concrete engineering work, while its score leaves room for improvement. section 8.2.1
  • Code-based graders are fast and reproducible but can reject valid variations. Model-based graders handle nuance but are nondeterministic and require calibration. Human graders support expert judgment and calibration but are costly and difficult to scale. section 8.2.1
Table 8.3 Comparison of grader types for AI agents across speed, cost, flexibility, and reproducibility
Grader type Advantages Disadvantages

Code-based

Fast, cheap, reproducible

Brittle, lacks nuance

Model-based

Flexible, scalable, handles open-ended tasks

Non-deterministic, requires calibration

Human

Best alignment with expert judgment

Expensive, slow, difficult to scale

Compare grader types by speed, cost, flexibility, and reproducibility when choosing how to score suite tasks. section 8.2.1

Evaluation in a RAG pipeline

Continuous telemetry can feed regression traces into automated evaluation suites. Separately, offline evaluation checks candidate retrieval indexes against golden test sets before deployment. Coverage validation checks whether evaluation datasets reflect evolving vocabulary, multi-hop queries, and real-world user intent. section 9.3.5

How Context Engineering covers it

How deep it goes
Defines suite components and grader trade-offs, and gives task-selection guidance for capability suites.
Examples
Uses challenging agent tasks, regression traces, and golden test sets for candidate retrieval indexes.
What you can do after
Evaluate and understand how to select capability cases and weigh grading approaches for an agent evaluation suite.
How it connects
Connects evaluation suites with evaluation harnesses, graders, transcripts, outcomes, regression testing, and retrieval evaluation.

Also explained in

Compare how 22 books explain test suite →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Context Engineering. The section links point to the parts of the book it draws on.