Chapter 15: Production RAG: Metrics, agentic systems, and continuous improvement
The introduction argues that production RAG performance depends heavily on how well the reader handles noisy retrieval, and distinguishes two ways readers respond to more context.
After this chapter you will be able to
- Reader Robustness to Noise. Explain why the reader's ability to filter irrelevant passages can matter more to answer quality than retriever quality, even when a new retriever improves recall.
- Two Context Scaling Patterns. Distinguish improve-then-plateau from peak-then-decline as more passages are retrieved, and explain why the reader's pattern affects how many documents to retrieve.
- Runtime technique selection. Distinguish fixed retrieval-technique choices from an agentic system’s runtime choice of which techniques to apply.
- Challenged retrieval assumptions. State which common assumptions about retrieval and answer quality the RAGGED framework tested, without claiming which specific assumptions failed.
- Reader responses to retrieval depth. Distinguish improve-then-plateau readers from peak-then-decline readers as the number of retrieved passages, k, increases.
- Retrieval depth as configuration. Justify tuning retrieval depth for a noise-sensitive reader rather than retrieving many passages just in case.
- Stability near optimal depth. Explain how the RAG Stability Score (RSS) measures performance around the optimal retrieval depth and why a low score signals a fragile production configuration.
- Value from additional passages. Explain how the RAG Scalability Coefficient (RSC) represents continued gains from retrieving more passages and what a high value indicates.
- Reader noise bottleneck. Explain why better retrieval does not always produce much better answers: even noise-sensitive readers use gold passages well when no irrelevant passages accompany them, but struggle when higher recall brings more noise.
- Fixed and agentic retrieval. Distinguish a predetermined RAG pipeline from agentic RAG, and explain how an agent can change its retrieval approach when results are poor, incomplete, or ambiguous.
- Reflection and answer revision. Explain how an agent reviews a generated answer against the query and retrieved context, then returns to retrieval or generation when it finds a gap.
- Planning across information sources. Describe how a planning agent divides a complex question into sub-queries, retrieves for each, and synthesizes the results.
- Runtime selection of tools. Explain how tool use lets an agent choose among databases, document retrieval, web search, APIs, or specialized retrievers according to the question.
- Specialized agent coordination tradeoff. Describe how specialized agents divide retrieval and generation tasks and explain the benefits and costs of coordinating them.
- Sequential prompt chaining. Describe how prompt chaining passes each step’s output to the next and when that predictable sequence is useful.
- Query-based pipeline routing. Explain how routing chooses a processing path from the incoming query, including the paths given for simple, complex, and ambiguous queries.
- Dynamic orchestrator-worker tasks. Distinguish orchestrator-workers from initial routing by explaining how an orchestrator assigns workers and can create new tasks in response to intermediate results.
- Evaluator-driven answer refinement. Trace how an evaluator’s score and feedback cause another generation attempt, and state when that loop ends.
- Agentic retrieval loop. Trace how the orchestrator takes a query and retrieval function, retrieves context, generates and evaluates a response, then either returns it or reformulates the query and tries again.
- Agentic accuracy latency tradeoff. Explain why extra evaluation and retrieval passes can make agentic RAG worthwhile for complex queries but wasteful for simple lookups.
- Autonomy and evidence quality. Distinguish an agent's ability to choose its next action from its ability to select good evidence, using the comparison with an engineered pipeline.
- Query complexity routing. Describe which kinds of queries warrant one retrieval pass, planning and decomposition, or iterative refinement.
- Shared ranking and generation skill. Explain why RankRAG treats context ranking and answer generation as related skills and how it combines them in one LLM.
- Shared ranking and generation. Explain why separately trained RAG components can be misaligned and how RankRAG changes the pipeline: retrieval remains separate, while one LLM ranks passages and generates an answer from the top-scoring ones.
- Ranking data and task synergy. Explain why a relatively small amount of ranking data can be effective in RankRAG’s instruction-tuning blend: answering from context and ranking passages reinforce each other.
- Mixed fine-tuning recipe. Describe the recommended addition to domain-specific generator fine-tuning and its expected pipeline benefit.
- Unified training example format. Describe how the section represents QA and ranking tasks as interleaved training examples.
- RAG and fine-tuning roles. Distinguish what RAG contributes from what fine-tuning contributes, and explain why a production system may use both.
- Domain-adapted retriever embeddings. Explain how fine-tuning an embedding model on domain data changes retrieval and which retrieval failures it targets.
- Noise-robust reader training. Describe how RAFT trains a reader to handle mixed relevant and irrelevant passages and why that can make retrieving more passages useful.
- Static versus self-improving RAG. Explain why a RAG pipeline configured before deployment may need repeated manual changes, and how a self-improving system differs.
- Learned Retrieval Influence Gate. Describe how the Meta-k Network uses retrieval signals to adjust the influence of neighbors on a prediction, and connect that idea to a RAG gate.
- Self-knowledge and retrieval value. Explain why SKILL-RAG considers what the model already knows when deciding which retrieved information to keep.
- Self-knowledge training and sentence filtering. Trace SKILL-RAG’s three steps from identifying known and unknown answers to filtering retrieved sentences at inference time.
- SimRAG two-stage training. Trace how SimRAG moves from general retrieval-oriented fine-tuning to self-generated domain QA pairs and continued fine-tuning.
- Round-trip consistency filter. Explain how SimRAG filters generated QA pairs and why that check relates to faithfulness and context relevance.
- Continuous improvement cycle. Describe the four stages of the production feedback loop—collect, evaluate, optimize, and deploy and measure—and explain why the cycle repeats.
- Production signals and diagnosis. Explain what production signals are collected and how the section uses RAGAS scores to distinguish a likely generation problem from a retrieval problem.
- Diagnosis-driven optimization choices. Match the section's example failure diagnoses to their proposed fixes, and describe what the named pipeline-level tools search or test.
- Complexity and payoff tradeoff. Explain why a simple retrieve-and-generate pipeline can be sufficient, and why added techniques should earn their latency, cost, and engineering effort.
- Failure modes by complexity. Distinguish retrieval-side gaps that a better single pass can address from harder problems that call for iteration, evaluation, or reasoning.
- Incremental technique selection. Describe the section's practical order for adding techniques and the evaluation-based stopping rule.
- Metric-led failure diagnosis. Describe how to begin diagnosing an underperforming RAG system: run the RAGAS evaluation triad, identify which metric is low, and use that result to follow a decision tree toward a failure point and a suitable technique.
- Query complexity and retrieval choice. Distinguish simple factual lookups from queries requiring multi-step reasoning, and describe the recommended starting retrieval approach for each.
- Reader sensitivity and retrieval depth. Describe how to profile the reader across retrieval depths and use its performance pattern to decide whether to retrieve more passages or improve context selection first.
- Evaluation baseline and priority. Explain when to establish a RAGAS-triad baseline and how its results guide the first optimization.
- Operational production gap. Explain why moving a working RAG prototype into production is primarily an operational challenge rather than a change to its retrieval and generation components.
- Retrieval context for diagnosis. Explain why production RAG needs continuous measurement and why each response should be logged with its full retrieval context.
- RAGAS trend interpretation. Describe how to sample production queries for RAGAS evaluation and what changes in context relevance or faithfulness can indicate.
- Whole Pipeline Versioning. Explain why the full RAG pipeline configuration should be treated as one versioned artifact rather than tracking components independently.
- Change Specific Reevaluation. Describe the checks the section calls for when an embedding model, corpus, or system prompt changes.
- Framework selection tradeoffs. Distinguish when LangChain, LlamaIndex, Haystack, and DSPy fit a RAG project, including the tradeoff each brings.
- Direct implementation fit. Explain when building a RAG pipeline directly is preferable to adopting a framework and what that choice involves.
- Component testing of pipeline stages. Explain what component tests check in a RAG pipeline and when to run them.
- End-to-end integration testing. Explain why integration tests check the full pipeline and when to run them.
- Production version regression testing. Describe how regression tests use past failure cases to decide whether a new pipeline version is safe to deploy.
- PDF parsing failure. Explain how corrupted text extraction from a PDF figure can cause both retrieval misses and fabricated answers.
- Cropped page-image context. Describe how sending a rendered figure region as an image helps the model read information that text extraction destroyed, and why the region matters.
- Two image insertion points. Distinguish adding a page image to generation context from building a visual retrieval index.
- Production modality choices. Apply the section's three-way guidance for choosing reranked text, generation-time page images, or a visual index.
- Prototype evaluation limits. Explain why the case study’s results should be read as prototype measurements rather than production results: the assistant was evaluated offline against a baseline using 85 questions and one 8-billion-parameter model.
- Ambiguous fintech documentation. Explain how fragmented documentation and team-specific terminology can cause a RAG system to retrieve the wrong document even when the needed content exists.
- Baseline pipeline and performance. Describe the B-RAG baseline, what it measured, and the kinds of questions on which it struggled.
- Orchestrated retrieval decisions. Explain how the orchestrator routes queries to specialized agents and how the intent classifier decides whether a follow-up needs retrieval or can use conversation history.
- Query vocabulary and acronym repair. Describe how query reformulation and acronym resolution make a user's wording more likely to retrieve the intended documents.
- Confidence-gated answer retry. Trace how the QA agent's assessment can send an answer back through retrieval.
- Retrieval gains and latency cost. Describe how A-RAG improved retrieval accuracy over the baseline while increasing latency, and explain why counting semantically equivalent retrievals raises the reported accuracy.
- Response quality score distribution. Describe how the LLM-as-judge scores compared A-RAG with B-RAG, including changes in low-quality and excellent responses.
- Sub-query generation gains. Explain why sub-query generation accounted for most of the gain, especially on vague procedural questions whose answers span multiple documents.
- Latency-based pipeline routing. Justify routing simple queries to a fast baseline and reserving the agentic pipeline for complex queries, given the observed latency and different user expectations.
- Resolved retrieval and answer failures. Explain how the baseline’s fragmented knowledge and acronym ambiguity led to FP1 and FP2, how A-RAG addressed them, and how the QA agent responded to FP7 incomplete answers.
- Remaining answer specificity failure. Distinguish the remaining FP6 failure from the retrieval failures: the system could give an overview when steps were wanted, or steps when an overview was wanted. Explain why the section identifies this as a generation-side problem outside the agentic retrieval pipeline.
- Reader Context Robustness. Explain how the agentic components improved answers both by finding better documents and by giving the generator cleaner, more focused context through sub-query decomposition and re-ranking.
- Selective Agentic Complexity. Explain why queries should be routed by complexity rather than all sent through the full agentic pipeline.
- Domain Specific Incremental Evaluation. Describe the recommended order for improving a similar system: establish an honest domain-specific evaluation, target measured failure points with agentic components, and measure each addition’s impact. Explain why the fintech team’s expert-validated question set mattered.
- Structural limits of RAG. Explain that some unsolved problems affect RAG as a paradigm rather than any one technique, and that recognizing this helps teams set realistic expectations and avoid over-engineering.
- Fragmented evaluation results. Explain why RAG evaluation results are difficult to compare across studies despite the availability of the RAGAS evaluation triad.
- Public benchmark mismatch. Distinguish typical public benchmark questions from enterprise RAG queries, and describe the proposed way to narrow that evaluation gap and its cost.
- Retrieval Corpus Attack Surface. Explain how an attacker-controlled document can affect a RAG system’s output when it enters the knowledge base and is retrieved.
- Confidential Retrieval and Permissions. Explain the privacy risk in enterprise RAG and why document-level access control remains challenging.
- Cumulative technique overhead. Explain how combining quality-enhancing RAG techniques increases latency and cost through additional retrieval or model-processing steps.
- Use-case quality latency balance. Justify choosing RAG techniques according to a production system’s latency budget and the quality improvement each technique provides for its use case. Describe how stability and scalability metrics can inform that choice without providing a universal optimum.
- Cross-source retrieval gap. Explain why indexing enterprise sources separately and routing a query to one index can fail when the answer is spread across systems.
- Corpus Aggregation Routing. Explain why passage retrieval cannot reliably answer corpus-wide arithmetic questions and what kind of tool should handle them.
What it builds on
- The correlation gap: When good retrieval metrics produce bad answers (chapter 14) · The gap between retrieval metrics and answer quality.
- Naive, advanced, modular, and agentic RAG (chapter 1) · Introduces the Naive, Advanced, and Modular RAG categories.
- The agentic spectrum (chapter 11) · Introduces degrees of agency in RAG systems.
- Putting the triad to work (chapter 14) · The RAGAS evaluation triad used to assess RAG failures.
- Types of compression (chapter 13) · Introduces context compression as an approach to reducing material passed to the generator.
- The concept of self-reflection (chapter 9) · Introduces Self-RAG’s self-evaluation of retrieved context.
- Corrective actions and web search integration (chapter 11) · Introduces corrective actions for low-quality retrieval.
- Reflection tokens for retrieval, relevance, and critique (chapter 9) · Self-RAG’s decisions about retrieval and critique.
Section by section
Introduction
The introduction argues that production RAG performance depends heavily on how well the reader handles noisy retrieval, and distinguishes two ways readers respond to more context.
- Reader Robustness to Noise: Explain why the reader's ability to filter irrelevant passages can matter more to answer quality than retriever quality, even when a new retriever improves recall.
Why it matters: It explains why improving retrieval alone may fail to improve a production system's answers. - Two Context Scaling Patterns: Distinguish improve-then-plateau from peak-then-decline as more passages are retrieved, and explain why the reader's pattern affects how many documents to retrieve.
Why it matters: The reader's response to additional context helps determine an appropriate retrieval count. - Stability Versus Scalability: Distinguish stable performance near a reader's optimal retrieval count from continued benefit as more context is added.
Why it matters: Consistent performance at a chosen retrieval count does not mean adding more context will help.
Key terms: reader robustness, noise, retriever quality, recall, improve-then-plateau, peak-then-decline, k, stability, scalability, optimal k
15.1 The evolving RAG landscape
The section traces RAG’s growing flexibility and complexity, then identifies runtime technique selection and evidence challenging common retrieval assumptions as two shifts in the field.
- RAG architecture progression: Describe how Naive, Advanced, and Modular RAG differ, and explain the accompanying increase in capability and complexity.
Why it matters: This progression explains why greater flexibility also brings more system complexity. - Runtime technique selection: Distinguish fixed retrieval-technique choices from an agentic system’s runtime choice of which techniques to apply.
Why it matters: Runtime selection is the key distinction this section draws between agentic RAG and fixed retrieval designs. - Challenged retrieval assumptions: State which common assumptions about retrieval and answer quality the RAGGED framework tested, without claiming which specific assumptions failed.
Why it matters: It cautions against assuming that retrieval improvements alone will improve the final answer.
Key terms: Naive RAG, Advanced RAG, Modular RAG, agentic system, runtime, retrieved context, intermediate results, RAGGED framework, retrievers, reader models
15.1.1 More retrieval is not always better
Retrieving more passages can either help or hurt RAG answer quality, depending on how the reader handles extra context.
- Reader responses to retrieval depth: Distinguish improve-then-plateau readers from peak-then-decline readers as the number of retrieved passages, k, increases.
Why it matters: Adding context is not automatically a safer choice for every reader model. - RAG Stability Score interpretation: Explain what the RAG Stability Score indicates about a reader near its best retrieval depth.
Why it matters: Stability shows whether a small change in retrieval depth is likely to disrupt performance. - Retrieval depth as configuration: Justify tuning retrieval depth for a noise-sensitive reader rather than retrieving many passages just in case.
Why it matters: Choosing k without checking the reader’s behavior can sacrifice quality or waste resources.
Key terms: retrieval depth, k, improve-then-plateau, peak-then-decline, noise-sensitive, RAG Stability Score, RSS, noise-sensitive reader, latency, tokens
15.1.2 Measuring stability and scalability
This section introduces RSS and RSC to measure how a RAG reader’s answer quality changes as retrieval depth varies.
- Stability near optimal depth: Explain how the RAG Stability Score (RSS) measures performance around the optimal retrieval depth and why a low score signals a fragile production configuration.
Why it matters: A reader that performs well only at one exact retrieval depth may be difficult to maintain when retrieval fluctuates. - Value from additional passages: Explain how the RAG Scalability Coefficient (RSC) represents continued gains from retrieving more passages and what a high value indicates.
Why it matters: RSC helps show whether adding passages remains useful, particularly for questions that draw on many sources. - Depth curve evaluation: Describe the measurements used to compute RSS and RSC, and distinguish what these metrics reveal from what the RAGAS evaluation triad reveals.
Why it matters: Scores collected across retrieval depths reveal sensitivity that an evaluation at one configuration could miss.
Key terms: RAG Stability Score, RSS, optimal k*, retrieval depth, RAG Scalability Coefficient, RSC, epsilon, performance curve, performance scores, answer quality
15.1.3 The reader is the bottleneck
RAGGED shows that a RAG reader’s ability to ignore irrelevant passages can limit the benefit of better retrieval, and compares ways to reduce that noise.
- Reader noise bottleneck: Explain why better retrieval does not always produce much better answers: even noise-sensitive readers use gold passages well when no irrelevant passages accompany them, but struggle when higher recall brings more noise.
Why it matters: It explains why a retrieval improvement may yield only a small gain in answer quality. - Approaches to noise tolerance: Describe how context compression, Self-RAG, and CRAG address noisy context in different ways, and explain that their shared aim here is to improve the reader’s effective noise tolerance rather than retrieval alone.
Why it matters: It points to interventions for cases where retrieving more material is not enough. - Reranking versus prompting: Distinguish the observed effects of reranking and simple noise-reduction prompting, and state why the section recommends prioritizing reranking.
Why it matters: It helps prioritize a more reliable intervention over a prompt-only attempt to handle noise.
Key terms: RAGGED, gold passages, noise-sensitive readers, recall, context compression, Self-RAG, CRAG, effective noise tolerance, peak-then-decline, improve-then-plateau
15.2 Agentic RAG: Autonomous retrieval systems
This section contrasts fixed RAG pipelines with agentic RAG, in which an LLM-powered agent adapts retrieval decisions at runtime.
- Fixed and agentic retrieval: Distinguish a predetermined RAG pipeline from agentic RAG, and explain how an agent can change its retrieval approach when results are poor, incomplete, or ambiguous.
Why it matters: The distinction explains why an agentic system can respond to retrieval failures that a fixed pipeline would simply pass along. - Agentic RAG in context: Explain how agentic RAG brings earlier adaptive retrieval ideas together, and describe how a coding assistant illustrates those decisions over a codebase.
Why it matters: It connects the agentic idea to both earlier RAG techniques and a familiar production use case.
Key terms: predetermined path, fixed pipeline, agentic RAG, autonomous agent, LLM-powered agent, Self-RAG, FLARE, CRAG, AI coding assistants
15.2.1 Four design patterns
This section explains four agentic RAG design patterns—reflection, planning, tool use, and multi-agent collaboration—and the coordination problem each addresses.
- Reflection and answer revision: Explain how an agent reviews a generated answer against the query and retrieved context, then returns to retrieval or generation when it finds a gap.
Why it matters: A first answer can miss or misstate evidence that retrieval already found. - Planning across information sources: Describe how a planning agent divides a complex question into sub-queries, retrieves for each, and synthesizes the results.
Why it matters: A single retrieved document may not contain every piece needed to answer a multi-hop question. - Runtime selection of tools: Explain how tool use lets an agent choose among databases, document retrieval, web search, APIs, or specialized retrievers according to the question.
Why it matters: Different questions call for different information sources, and some may need more than one. - Specialized agent coordination tradeoff: Describe how specialized agents divide retrieval and generation tasks and explain the benefits and costs of coordinating them.
Why it matters: Dividing work among agents can help, but the coordination itself becomes part of the system's cost and risk.
Key terms: reflection, retrieved context, FP5, FP7, planning, sub-queries, multi-hop questions, query expansion, FP1, tool use
15.2.2 From patterns to pipelines
The section explains how agentic RAG design patterns become pipelines that sequence steps, choose paths, assign dynamic tasks, and refine answers through evaluation.
- Sequential prompt chaining: Describe how prompt chaining passes each step’s output to the next and when that predictable sequence is useful.
Why it matters: Prompt chaining is the straightforward option when a query does not need to change the pipeline’s sequence. - Query-based pipeline routing: Explain how routing chooses a processing path from the incoming query, including the paths given for simple, complex, and ambiguous queries.
Why it matters: Routing avoids giving every query the same amount or kind of processing. - Dynamic orchestrator-worker tasks: Distinguish orchestrator-workers from initial routing by explaining how an orchestrator assigns workers and can create new tasks in response to intermediate results.
Why it matters: Dynamic task creation lets a pipeline respond to needs discovered while answering a query. - Evaluator-driven answer refinement: Trace how an evaluator’s score and feedback cause another generation attempt, and state when that loop ends.
Why it matters: The feedback loop gives a pipeline a way to revise weak answers before returning them.
Key terms: prompt chaining, focused prompt, routing, classifier, Adaptive-RAG, orchestrator-workers, orchestrator agent, worker agents, subtasks, evaluator-optimizer
15.2.3 Practical implementation
Shows how a simple orchestrator can run an agentic RAG loop that evaluates answers, retries retrieval for identified gaps, and limits retries.
- Agentic retrieval loop: Trace how the orchestrator takes a query and retrieval function, retrieves context, generates and evaluates a response, then either returns it or reformulates the query and tries again.
Why it matters: This loop adds reflection and targeted retries without requiring a complex agent framework. - Response evaluation signals: Distinguish the example's self-reported confidence and gap analysis from the RAGAS scores recommended for production.
Why it matters: A model's confidence is a convenient demonstration signal, but it is not the preferred production measure of answer quality. - Iteration limit and convergence: Explain why the loop caps iterations and describe the typical number of passes reported in the section.
Why it matters: A retry limit keeps an unanswerable query from consuming work indefinitely.
Key terms: agentic_rag_loop, retrieval function, _evaluate_response, _reformulate_query, gap analysis, self-reported confidence, RAGAS faithfulness, relevance scores, MAX_ITERATIONS, runaway loops
15.2.4 When agentic RAG helps and when it does not
This section weighs the benefits of agentic RAG against its latency, unnecessary searches, and evidence-quality limits, then relates its usefulness to query complexity.
- Agentic accuracy latency tradeoff: Explain why extra evaluation and retrieval passes can make agentic RAG worthwhile for complex queries but wasteful for simple lookups.
Why it matters: The extra work is valuable only when it improves an answer enough to justify the delay. - Unnecessary search incentives: Explain why a system controlling its own searches may keep retrieving after further searches cease to be useful.
Why it matters: Avoidable searches consume resources even when they do not improve the answer. - Autonomy and evidence quality: Distinguish an agent's ability to choose its next action from its ability to select good evidence, using the comparison with an engineered pipeline.
Why it matters: Letting a model orchestrate retrieval does not remove the need to evaluate the evidence it uses. - Query complexity routing: Describe which kinds of queries warrant one retrieval pass, planning and decomposition, or iterative refinement.
Why it matters: Matching the retrieval approach to the question avoids needless work while preserving help for harder queries.
Key terms: agentic RAG, latency, retrieval pass, search policies, outcome-based optimization, unnecessary search, agentic arm, engineer-drawn pipeline, query rewriting, evaluation
15.3 Fine-tuning in the RAG lifecycle
This section explains why fine-tuning and RAG can work together, using RankRAG’s shared approach to passage ranking and answer generation.
- RAG and fine-tuning roles: Distinguish RAG’s use of external evidence from fine-tuning’s adjustment of model parameters, and explain why treating them as alternatives is too restrictive.
Why it matters: The distinction helps explain why combining the two is possible. - Shared ranking and generation skill: Explain why RankRAG treats context ranking and answer generation as related skills and how it combines them in one LLM.
Why it matters: This shared skill is the reason RankRAG can combine ranking and generation rather than treating them as separate jobs.
Key terms: RAG, fine-tuning, external evidence, Self-RAG, RankRAG, context ranking, answer generation, instruction-tuning, query and context
15.3.1 RankRAG: Unifying ranking and generation
This section explains how RankRAG uses one instruction-tuned LLM to rank retrieved passages and generate answers, and how its training brings those tasks together.
- Shared ranking and generation: Explain why separately trained RAG components can be misaligned and how RankRAG changes the pipeline: retrieval remains separate, while one LLM ranks passages and generates an answer from the top-scoring ones.
Why it matters: It connects passage selection to the model that must use those passages to answer. - Two-stage training recipe: Describe how RankRAG first gains general instruction-following ability through supervised fine-tuning, then adds question-answering and passage-ranking data, including examples with gold and hard-negative passages.
Why it matters: The two stages explain how one model is prepared for both answering and passage selection. - Ranking data and task synergy: Explain why a relatively small amount of ranking data can be effective in RankRAG’s instruction-tuning blend: answering from context and ranking passages reinforce each other.
Why it matters: It explains why joint training can improve passage selection without relying on a large ranking-only dataset.
Key terms: RankRAG, reranker, generator, instruction-tuned LLM, top-scoring passages, supervised fine-tuning, Stage I, Stage II, context-rich QA, retrieval-augmented QA
15.3.2 Results and implications
This section reports RankRAG’s results and shows how to combine ranking and question-answering examples when fine-tuning a RAG generator.
- RankRAG results and transfer: Explain what the reported benchmarks show about RankRAG and why its ranking capability can transfer across domains.
Why it matters: The results support using general-purpose ranking data even when the application domain is specialized. - Mixed fine-tuning recipe: Describe the recommended addition to domain-specific generator fine-tuning and its expected pipeline benefit.
Why it matters: One fine-tuned model can handle both answer generation and relevance ranking. - Unified training example format: Describe how the section represents QA and ranking tasks as interleaved training examples.
Why it matters: A common example structure makes it possible to train one model on both relevance judgments and answers.
Key terms: RankRAG, query-passage relevance, generalization, domain-specific QA data, ranking examples, MS MARCO, reranker, (x, c, y), prompt, context, target, QA
15.3.3 Where fine-tuning fits in the RAG lifecycle
This section explains where fine-tuning helps a RAG system, from domain-specific retrieval and noise-resistant reading to practical adapter training and deployment.
- RAG and fine-tuning roles: Distinguish what RAG contributes from what fine-tuning contributes, and explain why a production system may use both.
Why it matters: This distinction helps you decide whether a failure calls for better access to information or better use of it. - Domain-adapted retriever embeddings: Explain how fine-tuning an embedding model on domain data changes retrieval and which retrieval failures it targets.
Why it matters: A retriever can miss useful material when its general-purpose notion of similarity does not fit the domain. - Noise-robust reader training: Describe how RAFT trains a reader to handle mixed relevant and irrelevant passages and why that can make retrieving more passages useful.
Why it matters: Retrieving more context helps only if the reader can separate useful evidence from distracting text. - Practical adapter fine-tuning: Explain why LoRA or QLoRA adapters make domain adaptation or reader robustness training more practical than full-model fine-tuning.
Why it matters: Adapters make the fine-tuning options in this section more feasible for teams with limited training resources. - Training and serving routes: Describe the local and managed routes from fine-tuning examples to a deployed model, including how adapters can be served.
Why it matters: A fine-tuning plan needs a workable path from training examples to an endpoint that serves the result.
Key terms: RAG, fine-tuning, retriever fine-tuning, embedding model, FP1, FP2, RAFT, reader fine-tuning, distractor passages, noise robustness
15.4 Self-improving RAG: The feedback loop
This section contrasts manually maintained RAG pipelines with self-improving systems that use their own signals to adapt as performance drifts.
- Static versus self-improving RAG: Explain why a RAG pipeline configured before deployment may need repeated manual changes, and how a self-improving system differs.
Why it matters: It explains the production problem that motivates a feedback loop.
Key terms: performance drifts, self-improving RAG system, own signals
15.4.1 The gatekeeper: Learning when to trust retrieval
This section uses adaptive nearest-neighbor machine translation to show how a learned gate can reduce harmful retrieval influence in RAG.
- Uneven Value of Retrieval: Explain why using a fixed number of retrieved neighbors for every token can hurt predictions as well as help them.
Why it matters: A gate is useful only if retrieval sometimes adds noise rather than useful evidence. - Learned Retrieval Influence Gate: Describe how the Meta-k Network uses retrieval signals to adjust the influence of neighbors on a prediction, and connect that idea to a RAG gate.
Why it matters: The gate offers a way to keep useful retrieval while suppressing results that would make an answer worse. - Gate Results and Transfer: State what the reported results show about the gate's usefulness, size, and transfer across domains.
Why it matters: The results support the practical claim that a small gate can learn useful signals without replacing each domain's datastore.
Key terms: fixed number of retrieved neighbors, kNN-MT, parametric knowledge, Meta-k Network, neighbor distances, distribution of values, learned gate, base model, BLEU, fixed-k baseline
15.4.2 The critic: Learning which memories to trust
This section explains how SKILL-RAG uses a model’s self-knowledge to keep useful retrieved sentences while filtering out redundant or misleading context.
- Self-knowledge and retrieval value: Explain why SKILL-RAG considers what the model already knows when deciding which retrieved information to keep.
Why it matters: More retrieved text is not necessarily more useful context. - Self-knowledge training and sentence filtering: Trace SKILL-RAG’s three steps from identifying known and unknown answers to filtering retrieved sentences at inference time.
Why it matters: The method turns a model’s estimate of its own knowledge into a sentence-level context filter. - Surgical filtering benefits: Describe how sentence-level filtering differs from a binary retrieval gate and why the section says it can improve RAG answers.
Why it matters: Filtering noise can make the remaining evidence easier for a reader model to use.
Key terms: SKILL-RAG, self-knowledge, retrieval, self-knowledge dataset, known, unknown, GRPO, Pointwise Mutual Information (PMI), positive PMI, gatekeeper
15.4.3 The student: Learning to teach itself to use retrieval
This section explains how SimRAG uses unlabeled domain documents to create and filter training examples that improve a model’s ability to use retrieval in specialized domains.
- Domain-specific RAG cold start: Explain why a general-purpose LLM may perform poorly with a specialized corpus and why ordinary fine-tuning is difficult to start.
Why it matters: This explains why SimRAG generates training examples from documents instead of relying on an existing labeled dataset. - SimRAG two-stage training: Trace how SimRAG moves from general retrieval-oriented fine-tuning to self-generated domain QA pairs and continued fine-tuning.
Why it matters: The two stages show how unlabeled domain text can become useful fine-tuning data. - Round-trip consistency filter: Explain how SimRAG filters generated QA pairs and why that check relates to faithfulness and context relevance.
Why it matters: The filter is what keeps self-generated practice examples tied to evidence the retriever can find.
Key terms: cold-start problem, domain-specific RAG, labeled QA pairs, SimRAG, Stage I, Stage II, unlabeled domain corpus, synthetic data, round-trip consistency check, top-k passages
15.4.4 Building the feedback loop
This section turns RAG evaluation and targeted fixes into a repeating production feedback loop.
- Continuous improvement cycle: Describe the four stages of the production feedback loop—collect, evaluate, optimize, and deploy and measure—and explain why the cycle repeats.
Why it matters: The loop makes production changes responsive to measured behavior rather than a one-time choice of technique. - Production signals and diagnosis: Explain what production signals are collected and how the section uses RAGAS scores to distinguish a likely generation problem from a retrieval problem.
Why it matters: A useful fix depends on diagnosing where the observed failure occurs. - Diagnosis-driven optimization choices: Match the section's example failure diagnoses to their proposed fixes, and describe what the named pipeline-level tools search or test.
Why it matters: Matching an intervention to its diagnosed failure avoids changing unrelated parts of the pipeline.
Key terms: feedback loop, collect, evaluate, optimize, deploy and measure, RAGAS triad, faithfulness, context relevance, failure points, user interactions
15.5 Practical frameworks for RAG technique selection
Introduces two frameworks for choosing RAG techniques by weighing complexity against performance and matching techniques to failure modes.
15.5.1 Complexity versus performance
This section shows how to choose RAG pipeline additions by weighing their complexity against the failure points they address and the measured improvement they deliver.
- Complexity and payoff tradeoff: Explain why a simple retrieve-and-generate pipeline can be sufficient, and why added techniques should earn their latency, cost, and engineering effort.
Why it matters: It keeps a production pipeline from becoming more costly and fragile without a corresponding benefit. - Failure modes by complexity: Distinguish retrieval-side gaps that a better single pass can address from harder problems that call for iteration, evaluation, or reasoning.
Why it matters: Matching a technique to the failure helps avoid trying to fix an answer-generation problem solely by tuning retrieval. - Incremental technique selection: Describe the section's practical order for adding techniques and the evaluation-based stopping rule.
Why it matters: This gives you a measurable stopping point instead of adding every available RAG technique.
Key terms: retrieve-and-generate pipeline, implementation complexity, failure points, single retrieval pass, hallucination, incomplete answers, multi-hop missing content, reader robustness, hybrid search, reranking
15.5.2 Mapping failure modes to RAG approaches
This section uses the RAGAS evaluation triad to diagnose RAG failures and choose remedies, including when failures occur together.
- Metric-led failure diagnosis: Describe how to begin diagnosing an underperforming RAG system: run the RAGAS evaluation triad, identify which metric is low, and use that result to follow a decision tree toward a failure point and a suitable technique.
Why it matters: A measured failure point gives you a reason to choose a particular remedy. - Compound failure diagnosis: Explain why one remedy may not be enough when failures combine, using the section's low-ranked document and hallucination example. Distinguish the diagnostic roles of context relevance, faithfulness, and answer relevance.
Why it matters: Separating overlapping failures helps avoid treating only one part of a production problem.
Key terms: RAGAS evaluation triad, failure point, decision tree, context relevance, faithfulness, answer relevance, reranking, noise robustness
15.5.3 Decision tree for new projects
Provides a decision sequence for choosing and revising a new RAG project's techniques based on its queries, reader model, evaluation results, and changing workload.
- Query complexity and retrieval choice: Distinguish simple factual lookups from queries requiring multi-step reasoning, and describe the recommended starting retrieval approach for each.
Why it matters: Matching retrieval to the queries avoids starting with techniques the use case does not need. - Reader sensitivity and retrieval depth: Describe how to profile the reader across retrieval depths and use its performance pattern to decide whether to retrieve more passages or improve context selection first.
Why it matters: More retrieved context helps only if the reader can handle it without losing performance. - Evaluation baseline and priority: Explain when to establish a RAGAS-triad baseline and how its results guide the first optimization.
Why it matters: A baseline directs effort toward the failure that matters most in the actual deployment. - Revisiting configuration as workloads change: Explain why technique selection must be revisited and identify the changes that can alter the best configuration.
Why it matters: A choice that fits today's workload may stop fitting after the data, queries, or models change.
Key terms: query distribution, naive RAG, reranking, multi-step retrieval, reader model, RAGGED's metrics, retrieval depth, noise-sensitive, noise-robust, context compression
15.6 Best practices for production deployment
This section introduces the operational work that separates a production RAG system from a working prototype.
- Operational production gap: Explain why moving a working RAG prototype into production is primarily an operational challenge rather than a change to its retrieval and generation components.
Why it matters: A working answer pipeline is not, by itself, a system ready for real traffic.
Key terms: RAG prototype, production system, monitoring, versioning, testing
15.6.1 Instrument everything
The section explains how to detect and diagnose silent production RAG failures by logging retrieval context and monitoring evaluation trends.
- Retrieval context for diagnosis: Explain why production RAG needs continuous measurement and why each response should be logged with its full retrieval context.
Why it matters: A query and answer alone may show that something went wrong, but not which part of the pipeline caused it. - RAGAS trend interpretation: Describe how to sample production queries for RAGAS evaluation and what changes in context relevance or faithfulness can indicate.
Why it matters: Score trends help point investigations toward retrieval or generation instead of treating every low score in isolation. - RSS stability monitoring: Explain what a declining RAGGED RSS score suggests and what pipeline conditions it may signal.
Why it matters: Stability monitoring can reveal a growing sensitivity to retrieval changes even as the corpus evolves.
Key terms: retrieval context, bad retrieval, bad generation, RAGAS evaluation triad, context relevance, faithfulness, RAGGED, RSS score, chunk size, retrieval depth
15.6.2 Version your pipeline as a unit
This section explains why a production RAG pipeline should be versioned as a whole, retested when its components change, and logged so regressions can be traced to a configuration.
- Whole Pipeline Versioning: Explain why the full RAG pipeline configuration should be treated as one versioned artifact rather than tracking components independently.
Why it matters: Versioning the whole configuration makes it possible to connect a change with the behavior it produced. - Change Specific Reevaluation: Describe the checks the section calls for when an embedding model, corpus, or system prompt changes.
Why it matters: These checks give teams a chance to find regressions before users do. - Manifest Based Regression Tracing: Explain how the version manifest supports response tracing and investigation of falling evaluation scores.
Why it matters: A logged manifest helps pinpoint which configuration change accompanied a regression.
Key terms: pipeline configuration, versioned artifact, embedding model, corpus, evaluation suite, evaluation dataset, RAGPipelineVersion, version manifest, to_dict, system_prompt_hash
15.6.3 RAG framework comparison
This section compares RAG frameworks and direct implementation by the control, abstractions, and pipeline complexity they offer.
- Framework selection tradeoffs: Distinguish when LangChain, LlamaIndex, Haystack, and DSPy fit a RAG project, including the tradeoff each brings.
Why it matters: Matching a framework to the job can save work without hiding the behavior you need to control. - Direct implementation fit: Explain when building a RAG pipeline directly is preferable to adopting a framework and what that choice involves.
Why it matters: A framework is not automatically the simpler choice, especially for a small or unusual pipeline.
Key terms: LangChain, LlamaIndex, Haystack, pipeline-as-DAG, DSPy, declarative program specification, Direct implementation, vector database clients, embedding APIs, LLM calls
15.6.4 The testing pyramid for RAG
This section explains how to test a RAG pipeline at component, integration, and regression levels, and how to keep the evaluation dataset useful as the system changes.
- Component testing of pipeline stages: Explain what component tests check in a RAG pipeline and when to run them.
Why it matters: Testing stages separately helps catch problems quickly as code changes. - End-to-end integration testing: Explain why integration tests check the full pipeline and when to run them.
Why it matters: A change can look harmless in an isolated metric yet alter the answer produced by the complete system. - Production version regression testing: Describe how regression tests use past failure cases to decide whether a new pipeline version is safe to deploy.
Why it matters: Previously troublesome queries provide a direct check against making known problems worse. - Living evaluation dataset: Explain how to maintain the evaluation dataset used for integration and regression tests.
Why it matters: The tests remain useful only if their examples reflect real failures and the current corpus.
Key terms: component tests, retriever, reranker, generator, integration tests, evaluation dataset, interaction effects, regression tests, previous production version, challenging queries
15.6.5 When the page is a picture
This section shows how image-based RAG can recover information lost by PDF parsing and how to choose between page images, visual retrieval, and a reranked text pipeline.
- PDF parsing failure: Explain how corrupted text extraction from a PDF figure can cause both retrieval misses and fabricated answers.
Why it matters: If information was lost during parsing, improving the retriever cannot bring it back. - Cropped page-image context: Describe how sending a rendered figure region as an image helps the model read information that text extraction destroyed, and why the region matters.
Why it matters: Giving the model the right pixels at readable resolution can fix an answer without changing the retriever. - Two image insertion points: Distinguish adding a page image to generation context from building a visual retrieval index.
Why it matters: These approaches solve different parts of the pipeline and have very different deployment costs. - Visual retrieval tradeoffs: Explain why visual retrieval's single-stage advantage does not make it the automatic default over reranked text retrieval.
Why it matters: A headline visual-retrieval score is not enough to justify the storage and operating cost of a production index. - Hybrid context tradeoff: Explain what combining visual pages and text passages gains, and when that combination can hurt.
Why it matters: Combining modalities can improve coverage without guaranteeing a better answer in every case. - Production modality choices: Apply the section's three-way guidance for choosing reranked text, generation-time page images, or a visual index.
Why it matters: Matching the intervention to the extent of parsing failure avoids rebuilding an index to fix a handful of pages.
Key terms: text extraction, character map, text index, page image, image block, keep argument, resolution, generation context, visual retrieval, rendered page
15.7 Case study: Agentic customer support in fintech
Introduces a fintech customer-support case study and frames its offline evaluation as prototype evidence, not a production result.
- Prototype evaluation limits: Explain why the case study’s results should be read as prototype measurements rather than production results: the assistant was evaluated offline against a baseline using 85 questions and one 8-billion-parameter model.
Why it matters: This distinction keeps the later case-study results in the context of what was actually tested.
Key terms: offline, baseline, prototype measurement, production result
15.7.1 The problem
The section explains why fragmented fintech documentation causes retrieval failures and establishes a measured baseline for the customer-support case study.
- Ambiguous fintech documentation: Explain how fragmented documentation and team-specific terminology can cause a RAG system to retrieve the wrong document even when the needed content exists.
Why it matters: It distinguishes a ranking problem from a missing-knowledge problem. - Baseline pipeline and performance: Describe the B-RAG baseline, what it measured, and the kinds of questions on which it struggled.
Why it matters: The baseline makes later changes comparable against a measured starting point.
Key terms: lexical ambiguity, FP2, missed the top rank, B-RAG, query reformulation agent, single-pass vector retrieval, summary agent, Hit Rate @5
15.7.2 The agentic design
This section explains how a fintech support A-RAG pipeline uses specialized agents to address retrieval failures, avoid unnecessary searches, and retry answers that score poorly.
- Orchestrated retrieval decisions: Explain how the orchestrator routes queries to specialized agents and how the intent classifier decides whether a follow-up needs retrieval or can use conversation history.
Why it matters: The system can address different problems without paying the latency cost of retrieval for every follow-up. - Query vocabulary and acronym repair: Describe how query reformulation and acronym resolution make a user's wording more likely to retrieve the intended documents.
Why it matters: These changes address both missed content and ambiguous terminology before retrieval. - Low-score sub-query recovery: Explain when the sub-query generator activates and how it uses targeted searches to recover evidence.
Why it matters: A weak initial search need not be the pipeline's last attempt to find relevant evidence. - Cross-encoder passage re-ranking: Describe what the re-ranker does to retrieved chunks and why its ranking differs from the initial retriever's.
Why it matters: Retrieving a relevant passage helps only if it is ranked where the answer process can use it. - Confidence-gated answer retry: Trace how the QA agent's assessment can send an answer back through retrieval.
Why it matters: A low-confidence answer prompts another evidence-gathering attempt rather than being returned without a retry.
Key terms: orchestrator, intent classifier, conversation history, query reformulator, acronym resolver, local glossary, FP1, FP2, sub-query generator, initial retrieval scores
15.7.3 Results
This section compares A-RAG with the baseline on retrieval accuracy, response quality, and latency, showing better results at a substantial speed cost.
- Retrieval gains and latency cost: Describe how A-RAG improved retrieval accuracy over the baseline while increasing latency, and explain why counting semantically equivalent retrievals raises the reported accuracy.
Why it matters: The retrieval improvement has to be weighed against a substantial latency cost. - Response quality score distribution: Describe how the LLM-as-judge scores compared A-RAG with B-RAG, including changes in low-quality and excellent responses.
Why it matters: The score distribution shows how often users received poor or excellent answers, not just the average quality. - Per-question preference and downside: Describe how often A-RAG was preferred, tied, or worse on individual questions, and characterize the reported size of its quality drops.
Why it matters: Per-question results show whether an average gain conceals frequent or severe regressions.
Key terms: strict retrieval accuracy, semantically equivalent retrievals, latency, LLM-as-judge, semantic accuracy scores, low-quality responses, excellent responses, per-question analysis, preferred, quality drop
15.7.4 What worked and what did not
The case study identifies which agentic components helped or hurt retrieval and why latency limits where the full pipeline can be deployed.
- Sub-query generation gains: Explain why sub-query generation accounted for most of the gain, especially on vague procedural questions whose answers span multiple documents.
Why it matters: It identifies the component that contributed most to this case study's improvement. - Acronym resolution limitations: Describe the mixed results of acronym resolution and why its implementation could reduce coverage.
Why it matters: Adding an agentic component does not guarantee a gain when its implementation is poorly matched to the corpus. - Re-ranking relevance tradeoffs: Explain when cross-encoder re-ranking helped and how it could hurt narrow queries despite improving relevance overall.
Why it matters: A passage that looks more relevant to a re-ranker is not always the one that answers the question. - Latency-based pipeline routing: Justify routing simple queries to a fast baseline and reserving the agentic pipeline for complex queries, given the observed latency and different user expectations.
Why it matters: The quality gains are useful in production only where response-time expectations allow them.
Key terms: sub-query generation, procedural questions, targeted retrievals, acronym resolution, coverage, heuristic regex matching, static glossary, cross-encoder re-ranking, relevance, latency
15.7.5 Connecting to the failure points
Maps the fintech support case study’s resolved and remaining failures to the RAG failure taxonomy.
- Resolved retrieval and answer failures: Explain how the baseline’s fragmented knowledge and acronym ambiguity led to FP1 and FP2, how A-RAG addressed them, and how the QA agent responded to FP7 incomplete answers.
Why it matters: The mapping shows which case-study failures were addressed by retrieval changes and which required another retrieval attempt. - Remaining answer specificity failure: Distinguish the remaining FP6 failure from the retrieval failures: the system could give an overview when steps were wanted, or steps when an overview was wanted. Explain why the section identifies this as a generation-side problem outside the agentic retrieval pipeline.
Why it matters: Finding information is not enough if the final answer is pitched at the wrong level of detail.
Key terms: FP1, FP2, FP7, A-RAG, sub-query generation, acronym resolution, FP6, incorrect specificity, generation-side problem, reader fine-tuning
15.7.6 Lessons for production teams
The fintech case study draws three production lessons: improve the generator’s context as well as retrieval, use agentic complexity selectively, and measure each change against domain-specific failures.
- Reader Context Robustness: Explain how the agentic components improved answers both by finding better documents and by giving the generator cleaner, more focused context through sub-query decomposition and re-ranking.
Why it matters: Production improvements should consider what the generator receives, not just which documents retrieval finds. - Selective Agentic Complexity: Explain why queries should be routed by complexity rather than all sent through the full agentic pipeline.
Why it matters: Selective routing preserves the benefits of agentic processing without imposing its cost on every query. - Domain Specific Incremental Evaluation: Describe the recommended order for improving a similar system: establish an honest domain-specific evaluation, target measured failure points with agentic components, and measure each addition’s impact. Explain why the fintech team’s expert-validated question set mattered.
Why it matters: Without a relevant baseline and measurements after each change, a team cannot tell which added complexity improves its system.
Key terms: reader robustness, sub-query decomposition, re-ranking, generator, query complexity, full agentic pipeline, average latency, domain-specific, internal evaluation dataset, domain experts
15.8 Current challenges and open research questions
This section frames RAG’s remaining unsolved problems as structural challenges and explains why recognizing them matters in practice.
- Structural limits of RAG: Explain that some unsolved problems affect RAG as a paradigm rather than any one technique, and that recognizing this helps teams set realistic expectations and avoid over-engineering.
Why it matters: This distinction helps you avoid promising that a more elaborate pipeline will solve a problem that remains open across RAG.
Key terms: structural challenges, RAG as a paradigm, over-engineering
15.8.1 Evaluation remains fragmented
This section explains why RAG evaluation results are hard to compare and why public benchmarks often fail to represent enterprise workloads.
- Fragmented evaluation results: Explain why RAG evaluation results are difficult to compare across studies despite the availability of the RAGAS evaluation triad.
Why it matters: A reported score may not measure the answer quality you care about or support a fair comparison with another system. - Public benchmark mismatch: Distinguish typical public benchmark questions from enterprise RAG queries, and describe the proposed way to narrow that evaluation gap and its cost.
Why it matters: Evaluation questions need to resemble the decisions and information needs the deployed system actually encounters.
Key terms: RAGAS, Exact Match, F1, lexical similarity, semantic correctness, NaturalQuestions, TriviaQA, enterprise RAG, synthetic evaluation data, corpus
15.8.2 Robustness and security
This subsection explains how the retrieval corpus creates security and privacy risks that current defenses do not fully address.
- Retrieval Corpus Attack Surface: Explain how an attacker-controlled document can affect a RAG system’s output when it enters the knowledge base and is retrieved.
Why it matters: Material retrieved as evidence can also become a route for attacker instructions. - Limits of Injection Defenses: Describe what indexing-time content filtering can catch and why it does not resolve prompt injection through retrieved passages.
Why it matters: Filtering a corpus helps, but it is not a complete security guarantee. - Confidential Retrieval and Permissions: Explain the privacy risk in enterprise RAG and why document-level access control remains challenging.
Why it matters: A relevant retrieved document must also be one the requester is permitted to see.
Key terms: retrieval corpus, prompt injection, poisoned document, content filtering, obfuscated, data privacy, access control, trust mechanisms
15.8.3 The efficiency-quality tradeoff
This subsection explains why quality-enhancing RAG techniques must be weighed against production latency budgets and why there is no universal way to choose the best configuration.
- Cumulative technique overhead: Explain how combining quality-enhancing RAG techniques increases latency and cost through additional retrieval or model-processing steps.
Why it matters: A quality gain can be outweighed by the time added when several techniques run on every query. - Use-case quality latency balance: Justify choosing RAG techniques according to a production system’s latency budget and the quality improvement each technique provides for its use case. Describe how stability and scalability metrics can inform that choice without providing a universal optimum.
Why it matters: A production answer must arrive soon enough to be useful, not merely score well on quality.
Key terms: latency, reranking, multi-step retrieval, agentic RAG, Self-RAG, context compression, latency budgets, RAGGED, stability and scalability metrics, quality-latency frontier
15.8.4 Retrieval across heterogeneous sources
This section explains why retrieving answers across disconnected enterprise sources remains difficult even when individual documents can be parsed and indexed.
- Cross-source retrieval gap: Explain why indexing enterprise sources separately and routing a query to one index can fail when the answer is spread across systems.
Why it matters: A system can retrieve well from each source individually and still miss an answer that requires evidence from several sources. - Graph and modality limits: Distinguish two partial advances from a solution to disconnected enterprise knowledge: similar RAG patterns across modalities, and Graph RAG's use of entity relationships.
Why it matters: It prevents mistaking a reusable RAG pattern or a graph-based approach for a complete answer to disconnected sources.
Key terms: enterprise knowledge, indexes, routing layer, Graph RAG, knowledge graph, modalities, entity relationships
15.8.5 Open questions
This section identifies unresolved RAG limitations, sketches research directions for addressing them, and recommends testing new techniques on your own workload.
- Retrieval Integrated Model Training: Distinguish training an LLM to retrieve during pre-training from attaching a retriever to a frozen model, and explain how differentiable pipelines could jointly optimize retrieval, reranking, and generation.
Why it matters: It separates promising training approaches from capabilities that current RAG pipelines already provide. - Privacy Across Organizational Boundaries: Explain the problem federated RAG aims to address and what remains unresolved.
Why it matters: Cross-organization retrieval must protect the underlying data, not merely produce useful answers. - Corpus Aggregation Routing: Explain why passage retrieval cannot reliably answer corpus-wide arithmetic questions and what kind of tool should handle them.
Why it matters: A plausible number drawn from a few retrieved rows can be wrong for the entire collection. - Evolving Knowledge Version Reasoning: Explain why answering questions about changing policies requires more than filtering documents by timestamp.
Why it matters: A retrieved policy passage may be genuine yet no longer authoritative. - Self Maintaining Knowledge Bases: Describe how a proposed self-building knowledge base differs from today's hand-maintained corpus.
Why it matters: The quality of retrieval depends on how the corpus itself is built and kept current. - Workload Specific Research Validation: Describe how to decide whether a promising new RAG technique helps your own application.
Why it matters: A research result is useful in production only if it improves the workload you actually have.
Key terms: continuous pre-training with retrieval, differentiable retrieval pipelines, end-to-end gradient-based optimization, federated RAG, privacy-preserving retrieval, federated search protocols, aggregation and arithmetic, top-5 retrieval, SQL query, analytics API
15.9 Summary
Concepts explained in this chapter
- query decompositionQuery decomposition is the technique of splitting a complex information need into smaller sub-queries that can be searched or answered separately.
- RAGGEDRAGGED is an evaluation framework for testing retrieval-augmented generation assumptions across retrievers, reader models, datasets, and retrieval conditions.
- agentic RAGAgentic RAG is a retrieval-augmented generation architecture in which runtime decisions control retrieval, reasoning, or tool actions based on system state and signals.
- visual retrievalA visual retriever is a retrieval component that finds relevant documents or pages from visual representations rather than text alone.
- peak-then-decline behaviorPeak-then-decline behavior is a performance pattern in which adding context improves results up to a point and then reduces them.
- rag stability scoreThe RAG Stability Score measures how consistently retrieval-augmented generation performance remains near its optimal retrieval depth.
- reflection patternA reflection pattern is an agentic control pattern in which a system evaluates its own output or intermediate results and revises them when needed.
- fine-tuningFine-tuning is further training of a pretrained model on task-specific or domain-specific data.
- routingA query router is a system component that analyzes a query and directs it to an appropriate retrieval or processing path.
- query rewritingQuery reformulation is the transformation of an existing search query into an alternative formulation intended to retrieve more suitable information.
- robustnessNoise robustness is the ability of a system to maintain performance when inputs or retrieved evidence contain irrelevant or corrupted information.
- retrieval depthRetrieval depth is the configured number or extent of candidate results considered during retrieval.
- ColPaliColPali is a vision-language retrieval model that represents document pages visually for late-interaction search.
- multimodal embedding modelA multimodal embedder is a model that maps inputs from multiple modalities, such as text and images, into vector representations.
- page imageA page image is a rendered visual representation of an entire document page.
- text pipelineA text pipeline is a retrieval workflow that represents and searches documents using extracted textual content.
- retrieval-augmented generationRetrieval-augmented generation (RAG) supplies retrieved external information to a language model before it generates a response.
- readerA reader model is a question-answering component that extracts or generates an answer from retrieved documents or passages.
- planningPlanning is an agentic workflow pattern in which a system constructs an action sequence before or during task execution.
- synthetic dataSynthetic data is artificially generated data produced to resemble or serve the role of real data.
- rag scalability coefficientThe RAG Scalability Coefficient measures how much additional retrieved context continues to improve model performance across retrieval depths.
- rerankingA reranker is a model or retrieval component that reorders candidate results according to estimated relevance.
- iterative retrievalIterative retrieval is a retrieval strategy that performs multiple retrieval steps, using intermediate results or reasoning to formulate subsequent searches.
- reader robustnessReader robustness is the ability of an answer-producing component to maintain answer quality despite variation or imperfections in retrieved context.
- test suiteA test suite is a collection of tests designed to assess specified behaviors or failure conditions in a system.
- temporal reasoningTemporal reasoning is reasoning that relates events, facts, or entities according to their times and chronological relationships.
- compactionContext compression reduces retrieved or prompt context while retaining information relevant to the query and generation task.
- similarity searchSimilarity search finds items whose representations are close to a query representation according to a similarity or distance function.
- RagasRAGAS is a framework for evaluating retrieval-augmented generation systems with metrics including faithfulness, answer relevance, and context relevance.
- RankRAGRankRAG is a retrieval-augmented model fine-tuned to rerank passages and generate answers within a single model.
- regressionA regression is a deterioration in system behavior caused by a change that previously improved or preserved performance.
- adapter methodAdapter-based fine-tuning adapts a pretrained model by training added parameter modules while leaving the base model weights largely frozen.
- retrieval rankingRanking is the ordering or filtering of retrieved candidates according to a relevance score, including ordering passages by relevance to a query.
- SimRAGSimRAG is a self-improving retrieval-augmented generation method that creates training questions from an unlabeled domain corpus and retains examples supported by retrieved context.
- self-ragSKILL-RAG is a retrieval-augmented generation technique that filters retrieved information according to the model's self-knowledge.
- context relevanceContext relevance is a metric measuring how much retrieved context is useful for answering the question.
- fidelityFaithfulness measures whether a generated response is supported by the retrieved context rather than introducing unsupported claims.
- accuracyAccuracy is the proportion of evaluated system outputs judged correct according to a specified criterion.
- latencyInference time is the time required for a trained model or system to produce an output for an input.
- continuous improvementA closed-loop system observes an intermediate or resulting state and uses that evaluation to decide subsequent actions.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
