Offline evaluation assesses experiment targets against previously collected datasets before deployment. It compares prompt versions, model configurations, or retrieval parameters without exposing users to the change. section 8.1 section 8.3.2
The evaluation pipeline accepts targets and a dataset, generates one response for every test-case and target combination, scores outputs with configured metrics, and returns aggregate and per-case comparisons. Repeated runs average non-deterministic model outputs; retrieval can be included to test a complete RAG pipeline. section 8.3.2
Use offline evaluation to:
- Catch regressions before deployment and compare candidate variants against a stable test suite. section 8.3
- Inspect per-test-case results rather than relying only on aggregate scores; a variant may improve scheduling answers while regressing on insurance questions. section 8.3.2
- Treat LLM-as-judge scores as comparative signals, not objective quality measurements; calibrate deployment decisions against human evaluation. section 8.3.2
Figure 8.3 Offline evaluation pipeline. Targets and a test dataset feed into the pipeline. For each test case and target combination, the pipeline generates a response through the Model Service, scores it with each configured metric, and aggregates results. The output includes per-target scores, per-test-case details for debugging, and a comparison with improvement percentages.

Trace the pipeline from targets and test data through model responses and metric scoring to aggregate, per-case, and improvement comparisons. section 8.3.2




