Online evaluation assesses AI outputs during production serving. Interaction traces flow to a parallel evaluation pipeline that scores and records response quality without blocking requests or gating responses, providing ongoing monitoring and observability. section 4.4.1
Design online evaluation around production constraints:
- Control cost: evaluating every request can make judge-token costs compound rapidly at production scale. section 4.4.1
- Expect incomplete signals: open-ended queries usually lack reference answers, while human feedback is delayed, sparse, and biased toward failures. section 4.4.1
- Prefer single-answer grading with binary or coarse outputs, such as whether a response appears helpful or stays on topic. section 4.4.1 section 4.5
- Maintain reliability through golden-set regression testing, score-distribution analysis, and human verification sampling. section 4.5
Production helpfulness check
A serving system sends interaction traces to a non-blocking judge pipeline, which records a binary answer to questions such as “Does this response appear helpful?” or “Does it stay on topic?” section 4.4.1
Table 4.6 Comparing online and offline evaluation contexts
|
Dimension
|
Online Evaluation
|
Offline Evaluation
|
|
Purpose
|
Production monitoring
|
Development and testing
|
|
Execution
|
Parallel to serving, non-blocking
|
Batch processing
|
|
Primary Constraints
|
Cost and data availability
|
Accuracy and comprehensiveness
|
|
Reference Data
|
Usually unavailable
|
Available, or constructible
|
|
Typical scale
|
Every request or a sampled subset
|
Hundreds to thousands
|
|
Model choice
|
Small, fast, lower cost models
|
Frontier models for maximum accuracy
|
Compare online and offline evaluation by purpose, execution model, reference data, scale, and judge-model choice. section 4.4.1


