A RAG system can produce a plausible answer even when it retrieved the wrong document or missed part of a question. A single-pass pipeline has no opportunity to catch that failure before responding. Continuous improvement closes the loop: observe a result, judge whether it is adequate, and use that judgment to choose the next action. The loop can run during an answer, during model training, or across production releases. section 11.1 section 15.4.4
The timescale determines what the system can change:
- At query time, Self-RAG can assess retrieved context and decide to regenerate, retrieve again, reformulate the query, or abstain. LevelRAG instead decomposes a question, searches, summarizes, verifies coverage, and supplements missing evidence until verification passes. section 9.6 section 11.3
- During training, Atlas uses the reader’s ability to produce the right answer as feedback for its retriever. Helpful documents receive higher retrieval scores without requiring human-labeled query–document pairs. section 6.2.1
- After deployment, log queries, contexts, responses, and user interactions; evaluate failures; apply targeted fixes; then deploy and measure again. Treat user signals cautiously: no negative click does not prove satisfaction. section 15.4.4
A compliance assistant’s evaluation loop
Each week, compare a random sample of production queries with a fixed synthetic test set. The fixed set makes changes comparable; the production sample exposes new question types. When a score falls below threshold, an engineer checks the flagged responses and either fixes the confirmed regression or adds the new pattern to the test set. In one iteration, a reranking fix raised context relevance but lowered faithfulness; a subsequent grounding prompt addressed that regression. Improvement requires checking multiple dimensions, not just celebrating a better retrieval score. section 14.6
Table 14.4 Evaluation scores across improvement iterations
|
Iteration
|
Context relevance
|
Faithfulness
|
Answer relevance
|
Primary diagnosis
|
|
Baseline (week 1)
|
0.65
|
0.91
|
0.78
|
FP2: retrieval ranking
|
|
After the reranking fix (week 3)
|
0.82
|
0.84
|
0.81
|
FP3: faithfulness regression
|
|
After the grounding prompt (week 5)
|
0.83
|
0.90
|
0.83
|
Stable: monitor
|
Compare the baseline, reranking fix, and grounding prompt to see how a gain in one metric can expose a regression in another. section 14.6
Choose the lightest loop that addresses the failure. Multi-pass critique and revision add latency, so a single-pass grounding constraint, evidence-alignment check, or refusal-first prompt may be preferable outside high-trust settings. For production changes, diagnose retrieval and generation separately: falling context relevance points toward retrieval, while falling faithfulness with strong context relevance points toward generation. section 9.6 section 15.4.4










































