liveBook · Manning

Monitoring

As explained in LLM Evaluation and Alignment, The Foundational Ideas

Online monitoring is the continuous observation of a deployed system’s behavior and operational metrics.

A system that performs well offline can still fail in production: evaluation scores may not match user needs, and new edge cases can emerge. Online monitoring continuously observes deployed behavior and operational signals so teams can detect degradation and improve the system. section 1.1.2 section 1.3.1

Monitoring complements rather than replaces offline evaluation and regression testing. Offline evaluation uses a curated set before deployment; regression tests check known capabilities before changes. Online monitoring observes actual production traffic, where reference answers are usually unavailable. section 6.4.1

For a retrieval-augmented generation (RAG) system, a practical monitoring loop can:

  • Sample production conversations for judge-based scoring, and track reference-free metrics such as faithfulness and answer relevance over time. section 6.4.1
  • Alert on significant metric drops and route low-confidence or high-risk outputs to human review. section 6.4.1
  • Inspect traces of the formulated query, retrieval method, returned documents, and subsequent reasoning to locate the failure. section 6.4.1
  • Combine scores with user feedback, user actions, error logs, and A/B tests; use the findings to adjust prompts, context, the agent harness, or model weights. section 1.3.1

Diagnosing a retrieval gap

A dashboard shows low context recall but high faithfulness. This points to a retrieval gap: the answer may follow the retrieved context even though relevant documents are missing. Inspect the trace to determine whether the system formulated the wrong query or retrieval failed to surface the right documents. section 6.4.1 section 6.5

Table 6.4 Interpreting metric combinations for RAG diagnosis
Faithfulness
Answer relevance
Recall
Precision
Diagnosis
High
High
High
High
Working well
Low
High
High
High
Hallucination beyond retrieved context
High
Low
High
High
Off-topic generation. Misaligned.
High
High
Low
High
Retrieval missing relevant documents
High
High
Low
Low
Retrieval is fundamentally misaligned with the query
Low
Low
Low
Low
Systemic failure

Use these metric combinations to guide diagnosis when the corresponding scores are available. Production traffic usually lacks reference answers, so not every metric will be available online. section 6.4.1

Treat monitoring scores as signals, not universal measures of quality. A language-model judge can surface degradation, but should not be the sole decision-maker for safety-critical or high-stakes decisions; combine it with other safeguards and human review. section 1.1.2 section 4.4.5

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Outlines sampling, scoring, alerting, review, and trace-based diagnosis without providing an implementation.
Examples
Uses RAG metric combinations and agentic traces to distinguish a malformed query from a retrieval failure.
What you can do after
Diagnose production RAG regressions using metric trends and traces, then route risky outputs for review.
How it connects
Connects online monitoring to offline evaluation, regression testing, retrieval-augmented generation, and alignment.

Also explained in

Compare how 51 books explain monitoring →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.