A system that performs well offline can still fail in production: evaluation scores may not match user needs, and new edge cases can emerge. Online monitoring continuously observes deployed behavior and operational signals so teams can detect degradation and improve the system. section 1.1.2 section 1.3.1
Monitoring complements rather than replaces offline evaluation and regression testing. Offline evaluation uses a curated set before deployment; regression tests check known capabilities before changes. Online monitoring observes actual production traffic, where reference answers are usually unavailable. section 6.4.1
For a retrieval-augmented generation (RAG) system, a practical monitoring loop can:
- Sample production conversations for judge-based scoring, and track reference-free metrics such as faithfulness and answer relevance over time. section 6.4.1
- Alert on significant metric drops and route low-confidence or high-risk outputs to human review. section 6.4.1
- Inspect traces of the formulated query, retrieval method, returned documents, and subsequent reasoning to locate the failure. section 6.4.1
- Combine scores with user feedback, user actions, error logs, and A/B tests; use the findings to adjust prompts, context, the agent harness, or model weights. section 1.3.1
Diagnosing a retrieval gap
A dashboard shows low context recall but high faithfulness. This points to a retrieval gap: the answer may follow the retrieved context even though relevant documents are missing. Inspect the trace to determine whether the system formulated the wrong query or retrieval failed to surface the right documents. section 6.4.1 section 6.5
Table 6.4 Interpreting metric combinations for RAG diagnosis
|
Faithfulness
|
Answer relevance
|
Recall
|
Precision
|
Diagnosis
|
|
High
|
High
|
High
|
High
|
Working well
|
|
Low
|
High
|
High
|
High
|
Hallucination beyond retrieved context
|
|
High
|
Low
|
High
|
High
|
Off-topic generation. Misaligned.
|
|
High
|
High
|
Low
|
High
|
Retrieval missing relevant documents
|
|
High
|
High
|
Low
|
Low
|
Retrieval is fundamentally misaligned with the query
|
|
Low
|
Low
|
Low
|
Low
|
Systemic failure
|
Use these metric combinations to guide diagnosis when the corresponding scores are available. Production traffic usually lacks reference answers, so not every metric will be available online. section 6.4.1
Treat monitoring scores as signals, not universal measures of quality. A language-model judge can surface degradation, but should not be the sole decision-maker for safety-critical or high-stakes decisions; combine it with other safeguards and human review. section 1.1.2 section 4.4.5


















































