liveBook · Manning

Continuous improvement

As explained in Retrieval Augmented Generation, The Foundational Ideas

A closed-loop system observes an intermediate or resulting state and uses that evaluation to decide subsequent actions.

A RAG system can produce a plausible answer even when it retrieved the wrong document or missed part of a question. A single-pass pipeline has no opportunity to catch that failure before responding. Continuous improvement closes the loop: observe a result, judge whether it is adequate, and use that judgment to choose the next action. The loop can run during an answer, during model training, or across production releases. section 11.1 section 15.4.4

The timescale determines what the system can change:

  • At query time, Self-RAG can assess retrieved context and decide to regenerate, retrieve again, reformulate the query, or abstain. LevelRAG instead decomposes a question, searches, summarizes, verifies coverage, and supplements missing evidence until verification passes. section 9.6 section 11.3
  • During training, Atlas uses the reader’s ability to produce the right answer as feedback for its retriever. Helpful documents receive higher retrieval scores without requiring human-labeled query–document pairs. section 6.2.1
  • After deployment, log queries, contexts, responses, and user interactions; evaluate failures; apply targeted fixes; then deploy and measure again. Treat user signals cautiously: no negative click does not prove satisfaction. section 15.4.4

A compliance assistant’s evaluation loop

Each week, compare a random sample of production queries with a fixed synthetic test set. The fixed set makes changes comparable; the production sample exposes new question types. When a score falls below threshold, an engineer checks the flagged responses and either fixes the confirmed regression or adds the new pattern to the test set. In one iteration, a reranking fix raised context relevance but lowered faithfulness; a subsequent grounding prompt addressed that regression. Improvement requires checking multiple dimensions, not just celebrating a better retrieval score. section 14.6

Table 14.4 Evaluation scores across improvement iterations
Iteration
Context relevance
Faithfulness
Answer relevance
Primary diagnosis
Baseline (week 1)
0.65
0.91
0.78
FP2: retrieval ranking
After the reranking fix (week 3)
0.82
0.84
0.81
FP3: faithfulness regression
After the grounding prompt (week 5)
0.83
0.90
0.83
Stable: monitor

Compare the baseline, reranking fix, and grounding prompt to see how a gain in one metric can expose a regression in another. section 14.6

Choose the lightest loop that addresses the failure. Multi-pass critique and revision add latency, so a single-pass grounding constraint, evidence-alignment check, or refusal-first prompt may be preferable outside high-trust settings. For production changes, diagnose retrieval and generation separately: falling context relevance points toward retrieval, while falling faithfulness with strong context relevance points toward generation. section 9.6 section 15.4.4

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Details query-time and production-loop steps, plus reader-guided retriever training; leaves loop selection dependent on failure diagnosis and latency.
Examples
Uses a compliance assistant’s weekly tests and reranking-to-grounding-prompt iterations to expose metric tradeoffs.
What it assumes
Assumes familiarity with retrieval and generation pipelines, while explaining why a single-pass pipeline cannot correct intermediate failures.
What you can do after
Operate/configure a RAG feedback cycle by collecting signals, diagnosing failures, applying targeted fixes, and checking for regressions.
How it connects
Links agentic RAG, Self-RAG, LevelRAG, retriever training, RAGAS, and production evaluation.

Also explained in

Compare how 43 books explain continuous improvement →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.