liveBook · Manning

Online evaluation

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Online evaluation is the assessment of an AI system’s outputs during production serving without blocking user requests or gating responses.

Online evaluation assesses AI outputs during production serving. Interaction traces flow to a parallel evaluation pipeline that scores and records response quality without blocking requests or gating responses, providing ongoing monitoring and observability. section 4.4.1

Design online evaluation around production constraints:

  • Control cost: evaluating every request can make judge-token costs compound rapidly at production scale. section 4.4.1
  • Expect incomplete signals: open-ended queries usually lack reference answers, while human feedback is delayed, sparse, and biased toward failures. section 4.4.1
  • Prefer single-answer grading with binary or coarse outputs, such as whether a response appears helpful or stays on topic. section 4.4.1 section 4.5
  • Maintain reliability through golden-set regression testing, score-distribution analysis, and human verification sampling. section 4.5

Production helpfulness check

A serving system sends interaction traces to a non-blocking judge pipeline, which records a binary answer to questions such as “Does this response appear helpful?” or “Does it stay on topic?” section 4.4.1

Table 4.6 Comparing online and offline evaluation contexts
Dimension
Online Evaluation
Offline Evaluation
Purpose
Production monitoring
Development and testing
Execution
Parallel to serving, non-blocking
Batch processing
Primary Constraints
Cost and data availability
Accuracy and comprehensiveness
Reference Data
Usually unavailable
Available, or constructible
Typical scale
Every request or a sampled subset
Hundreds to thousands
Model choice
Small, fast, lower cost models
Frontier models for maximum accuracy

Compare online and offline evaluation by purpose, execution model, reference data, scale, and judge-model choice. section 4.4.1

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Explains execution constraints, judge-output choices, missing signals, and monitoring practices, but defers detailed judge implementation.
Examples
Uses production interaction traces and binary helpfulness or topicality judgments as monitoring artifacts.
What it assumes
Assumes production serving and evaluation pipelines, while establishing the distinction between observation and request-gating decisions.
What you can do after
Operate a non-blocking monitoring pipeline using low-cost single-answer judgments and ongoing calibration checks.
How it connects
Connects production serving to offline evaluation, content moderation, quality gates, routing systems, and LLM-as-a-judge.

Also explained in

Compare how 3 books explain online evaluation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.