liveBook · Manning

Test suite

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A test suite is an organized collection of tests and expected outcomes used to assess software or generated code.

An evaluation suite is an organized set of test cases and expected outcomes used to check whether a system behaves as intended. A stable regression suite reruns known cases after a change to detect capabilities that have degraded. section 6.4.1 section 8.4.4

For retrieval-augmented generation (RAG), an offline suite can contain queries, relevant documents, and reference answers. It can measure retrieval and answer quality before changes reach users. A smaller regression set can run before each change; drops beyond agreed thresholds trigger release review or blocking. Evaluate retrieval quality before interpreting generation scores, because poor retrieval can depress those scores even when generation is working correctly. section 6.4.1 section 6.5

A nightly regression signal

A CI/CD pipeline can run 500 query-response pairs each night. A 5% drop in ROUGE-L after a prompt-template change signals that output has changed relative to the baseline. The score does not explain the semantic change, so it prompts human review or targeted model-based evaluation. section 2.3.3

To turn a behavior specification into an executable suite:

  • Translate each clause into prompts and checks for the required behavior. For example, check whether numerical claims have citations, or whether the assistant says it does not know when an answer is absent from retrieved context. section 8.4.4
  • Add a test for a specific production misbehavior to the regression suite, and evaluate the next training run against it. section 8.4.4
  • Version the specification and run the suite on training runs, specification changes, and deployments. A failing suite blocks deployment. section 8.5.4

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains how to derive tests from clauses, add regression cases, set review thresholds, and use suite results as a deployment gate.
Examples
Uses a 500-pair nightly RAG suite and checks for citations and admission of missing answers.
What you can do after
Operate/configure it by turning specification clauses into checks, adding failure cases, and using results to gate deployments.
How it connects
Connects regression testing with RAG evaluation, continuous integration, and behavioral specifications.

Also explained in

Compare how 22 books explain test suite →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.