liveBook · Manning

Offline evaluation

As explained in Designing AI Systems

Offline evaluation assesses an AI system or change using previously collected datasets rather than live interactions.

Offline evaluation assesses experiment targets against previously collected datasets before deployment. It compares prompt versions, model configurations, or retrieval parameters without exposing users to the change. section 8.1 section 8.3.2

The evaluation pipeline accepts targets and a dataset, generates one response for every test-case and target combination, scores outputs with configured metrics, and returns aggregate and per-case comparisons. Repeated runs average non-deterministic model outputs; retrieval can be included to test a complete RAG pipeline. section 8.3.2

Use offline evaluation to:

  • Catch regressions before deployment and compare candidate variants against a stable test suite. section 8.3
  • Inspect per-test-case results rather than relying only on aggregate scores; a variant may improve scheduling answers while regressing on insurance questions. section 8.3.2
  • Treat LLM-as-judge scores as comparative signals, not objective quality measurements; calibrate deployment decisions against human evaluation. section 8.3.2
Figure 8.3 Offline evaluation pipeline. Targets and a test dataset feed into the pipeline. For each test case and target combination, the pipeline generates a response through the Model Service, scores it with each configured metric, and aggregates results. The output includes per-target scores, per-test-case details for debugging, and a comparison with improvement percentages.

Trace the pipeline from targets and test data through model responses and metric scoring to aggregate, per-case, and improvement comparisons. section 8.3.2

How Designing AI Systems covers it

How deep it goes
Details repeated runs, retrieval-inclusive testing, aggregate scores, per-case diagnostics, and LLM-judge limitations; it stops short of absolute quality measurement.
Examples
Contrasts helpfulness scores of 0.82 and 0.78, then diagnoses scheduling improvements alongside insurance regressions.
What it assumes
Assumes experiment targets, evaluation datasets, configured metrics, and model execution; teaches how they form a pre-deployment comparison pipeline.
What you can do after
Evaluate candidate targets, identify regressions by test case, and decide whether changes merit promotion before production exposure.
How it connects
Connects evaluation datasets, prompt versions, model configurations, retrieval, online evaluation, human evaluation, observability, and A/B testing.

Also explained in

Compare how 5 books explain offline evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.