liveBook · Manning

Chapter 15: Production RAG: Metrics, agentic systems, and continuous improvement

What you will learn in chapter 15 of Retrieval Augmented Generation, The Foundational Ideas

The introduction argues that production RAG performance depends heavily on how well the reader handles noisy retrieval, and distinguishes two ways readers respond to more context.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction argues that production RAG performance depends heavily on how well the reader handles noisy retrieval, and distinguishes two ways readers respond to more context.

Key terms: reader robustness, noise, retriever quality, recall, improve-then-plateau, peak-then-decline, k, stability, scalability, optimal k

15.1 The evolving RAG landscape

The section traces RAG’s growing flexibility and complexity, then identifies runtime technique selection and evidence challenging common retrieval assumptions as two shifts in the field.

Key terms: Naive RAG, Advanced RAG, Modular RAG, agentic system, runtime, retrieved context, intermediate results, RAGGED framework, retrievers, reader models

15.1.1 More retrieval is not always better

Retrieving more passages can either help or hurt RAG answer quality, depending on how the reader handles extra context.

Key terms: retrieval depth, k, improve-then-plateau, peak-then-decline, noise-sensitive, RAG Stability Score, RSS, noise-sensitive reader, latency, tokens

15.1.2 Measuring stability and scalability

This section introduces RSS and RSC to measure how a RAG reader’s answer quality changes as retrieval depth varies.

Key terms: RAG Stability Score, RSS, optimal k*, retrieval depth, RAG Scalability Coefficient, RSC, epsilon, performance curve, performance scores, answer quality

15.1.3 The reader is the bottleneck

RAGGED shows that a RAG reader’s ability to ignore irrelevant passages can limit the benefit of better retrieval, and compares ways to reduce that noise.

Key terms: RAGGED, gold passages, noise-sensitive readers, recall, context compression, Self-RAG, CRAG, effective noise tolerance, peak-then-decline, improve-then-plateau

15.2 Agentic RAG: Autonomous retrieval systems

This section contrasts fixed RAG pipelines with agentic RAG, in which an LLM-powered agent adapts retrieval decisions at runtime.

Key terms: predetermined path, fixed pipeline, agentic RAG, autonomous agent, LLM-powered agent, Self-RAG, FLARE, CRAG, AI coding assistants

15.2.1 Four design patterns

This section explains four agentic RAG design patterns—reflection, planning, tool use, and multi-agent collaboration—and the coordination problem each addresses.

Key terms: reflection, retrieved context, FP5, FP7, planning, sub-queries, multi-hop questions, query expansion, FP1, tool use

15.2.2 From patterns to pipelines

The section explains how agentic RAG design patterns become pipelines that sequence steps, choose paths, assign dynamic tasks, and refine answers through evaluation.

Key terms: prompt chaining, focused prompt, routing, classifier, Adaptive-RAG, orchestrator-workers, orchestrator agent, worker agents, subtasks, evaluator-optimizer

15.2.3 Practical implementation

Shows how a simple orchestrator can run an agentic RAG loop that evaluates answers, retries retrieval for identified gaps, and limits retries.

Key terms: agentic_rag_loop, retrieval function, _evaluate_response, _reformulate_query, gap analysis, self-reported confidence, RAGAS faithfulness, relevance scores, MAX_ITERATIONS, runaway loops

15.2.4 When agentic RAG helps and when it does not

This section weighs the benefits of agentic RAG against its latency, unnecessary searches, and evidence-quality limits, then relates its usefulness to query complexity.

Key terms: agentic RAG, latency, retrieval pass, search policies, outcome-based optimization, unnecessary search, agentic arm, engineer-drawn pipeline, query rewriting, evaluation

15.3 Fine-tuning in the RAG lifecycle

This section explains why fine-tuning and RAG can work together, using RankRAG’s shared approach to passage ranking and answer generation.

Key terms: RAG, fine-tuning, external evidence, Self-RAG, RankRAG, context ranking, answer generation, instruction-tuning, query and context

15.3.1 RankRAG: Unifying ranking and generation

This section explains how RankRAG uses one instruction-tuned LLM to rank retrieved passages and generate answers, and how its training brings those tasks together.

Key terms: RankRAG, reranker, generator, instruction-tuned LLM, top-scoring passages, supervised fine-tuning, Stage I, Stage II, context-rich QA, retrieval-augmented QA

15.3.2 Results and implications

This section reports RankRAG’s results and shows how to combine ranking and question-answering examples when fine-tuning a RAG generator.

Key terms: RankRAG, query-passage relevance, generalization, domain-specific QA data, ranking examples, MS MARCO, reranker, (x, c, y), prompt, context, target, QA

15.3.3 Where fine-tuning fits in the RAG lifecycle

This section explains where fine-tuning helps a RAG system, from domain-specific retrieval and noise-resistant reading to practical adapter training and deployment.

Key terms: RAG, fine-tuning, retriever fine-tuning, embedding model, FP1, FP2, RAFT, reader fine-tuning, distractor passages, noise robustness

15.4 Self-improving RAG: The feedback loop

This section contrasts manually maintained RAG pipelines with self-improving systems that use their own signals to adapt as performance drifts.

Key terms: performance drifts, self-improving RAG system, own signals

15.4.1 The gatekeeper: Learning when to trust retrieval

This section uses adaptive nearest-neighbor machine translation to show how a learned gate can reduce harmful retrieval influence in RAG.

Key terms: fixed number of retrieved neighbors, kNN-MT, parametric knowledge, Meta-k Network, neighbor distances, distribution of values, learned gate, base model, BLEU, fixed-k baseline

15.4.2 The critic: Learning which memories to trust

This section explains how SKILL-RAG uses a model’s self-knowledge to keep useful retrieved sentences while filtering out redundant or misleading context.

Key terms: SKILL-RAG, self-knowledge, retrieval, self-knowledge dataset, known, unknown, GRPO, Pointwise Mutual Information (PMI), positive PMI, gatekeeper

15.4.3 The student: Learning to teach itself to use retrieval

This section explains how SimRAG uses unlabeled domain documents to create and filter training examples that improve a model’s ability to use retrieval in specialized domains.

Key terms: cold-start problem, domain-specific RAG, labeled QA pairs, SimRAG, Stage I, Stage II, unlabeled domain corpus, synthetic data, round-trip consistency check, top-k passages

15.4.4 Building the feedback loop

This section turns RAG evaluation and targeted fixes into a repeating production feedback loop.

Key terms: feedback loop, collect, evaluate, optimize, deploy and measure, RAGAS triad, faithfulness, context relevance, failure points, user interactions

15.5 Practical frameworks for RAG technique selection

Introduces two frameworks for choosing RAG techniques by weighing complexity against performance and matching techniques to failure modes.

15.5.1 Complexity versus performance

This section shows how to choose RAG pipeline additions by weighing their complexity against the failure points they address and the measured improvement they deliver.

Key terms: retrieve-and-generate pipeline, implementation complexity, failure points, single retrieval pass, hallucination, incomplete answers, multi-hop missing content, reader robustness, hybrid search, reranking

15.5.2 Mapping failure modes to RAG approaches

This section uses the RAGAS evaluation triad to diagnose RAG failures and choose remedies, including when failures occur together.

Key terms: RAGAS evaluation triad, failure point, decision tree, context relevance, faithfulness, answer relevance, reranking, noise robustness

15.5.3 Decision tree for new projects

Provides a decision sequence for choosing and revising a new RAG project's techniques based on its queries, reader model, evaluation results, and changing workload.

Key terms: query distribution, naive RAG, reranking, multi-step retrieval, reader model, RAGGED's metrics, retrieval depth, noise-sensitive, noise-robust, context compression

15.6 Best practices for production deployment

This section introduces the operational work that separates a production RAG system from a working prototype.

Key terms: RAG prototype, production system, monitoring, versioning, testing

15.6.1 Instrument everything

The section explains how to detect and diagnose silent production RAG failures by logging retrieval context and monitoring evaluation trends.

Key terms: retrieval context, bad retrieval, bad generation, RAGAS evaluation triad, context relevance, faithfulness, RAGGED, RSS score, chunk size, retrieval depth

15.6.2 Version your pipeline as a unit

This section explains why a production RAG pipeline should be versioned as a whole, retested when its components change, and logged so regressions can be traced to a configuration.

Key terms: pipeline configuration, versioned artifact, embedding model, corpus, evaluation suite, evaluation dataset, RAGPipelineVersion, version manifest, to_dict, system_prompt_hash

15.6.3 RAG framework comparison

This section compares RAG frameworks and direct implementation by the control, abstractions, and pipeline complexity they offer.

Key terms: LangChain, LlamaIndex, Haystack, pipeline-as-DAG, DSPy, declarative program specification, Direct implementation, vector database clients, embedding APIs, LLM calls

15.6.4 The testing pyramid for RAG

This section explains how to test a RAG pipeline at component, integration, and regression levels, and how to keep the evaluation dataset useful as the system changes.

Key terms: component tests, retriever, reranker, generator, integration tests, evaluation dataset, interaction effects, regression tests, previous production version, challenging queries

15.6.5 When the page is a picture

This section shows how image-based RAG can recover information lost by PDF parsing and how to choose between page images, visual retrieval, and a reranked text pipeline.

Key terms: text extraction, character map, text index, page image, image block, keep argument, resolution, generation context, visual retrieval, rendered page

15.7 Case study: Agentic customer support in fintech

Introduces a fintech customer-support case study and frames its offline evaluation as prototype evidence, not a production result.

Key terms: offline, baseline, prototype measurement, production result

15.7.1 The problem

The section explains why fragmented fintech documentation causes retrieval failures and establishes a measured baseline for the customer-support case study.

Key terms: lexical ambiguity, FP2, missed the top rank, B-RAG, query reformulation agent, single-pass vector retrieval, summary agent, Hit Rate @5

15.7.2 The agentic design

This section explains how a fintech support A-RAG pipeline uses specialized agents to address retrieval failures, avoid unnecessary searches, and retry answers that score poorly.

Key terms: orchestrator, intent classifier, conversation history, query reformulator, acronym resolver, local glossary, FP1, FP2, sub-query generator, initial retrieval scores

15.7.3 Results

This section compares A-RAG with the baseline on retrieval accuracy, response quality, and latency, showing better results at a substantial speed cost.

Key terms: strict retrieval accuracy, semantically equivalent retrievals, latency, LLM-as-judge, semantic accuracy scores, low-quality responses, excellent responses, per-question analysis, preferred, quality drop

15.7.4 What worked and what did not

The case study identifies which agentic components helped or hurt retrieval and why latency limits where the full pipeline can be deployed.

Key terms: sub-query generation, procedural questions, targeted retrievals, acronym resolution, coverage, heuristic regex matching, static glossary, cross-encoder re-ranking, relevance, latency

15.7.5 Connecting to the failure points

Maps the fintech support case study’s resolved and remaining failures to the RAG failure taxonomy.

Key terms: FP1, FP2, FP7, A-RAG, sub-query generation, acronym resolution, FP6, incorrect specificity, generation-side problem, reader fine-tuning

15.7.6 Lessons for production teams

The fintech case study draws three production lessons: improve the generator’s context as well as retrieval, use agentic complexity selectively, and measure each change against domain-specific failures.

Key terms: reader robustness, sub-query decomposition, re-ranking, generator, query complexity, full agentic pipeline, average latency, domain-specific, internal evaluation dataset, domain experts

15.8 Current challenges and open research questions

This section frames RAG’s remaining unsolved problems as structural challenges and explains why recognizing them matters in practice.

Key terms: structural challenges, RAG as a paradigm, over-engineering

15.8.1 Evaluation remains fragmented

This section explains why RAG evaluation results are hard to compare and why public benchmarks often fail to represent enterprise workloads.

Key terms: RAGAS, Exact Match, F1, lexical similarity, semantic correctness, NaturalQuestions, TriviaQA, enterprise RAG, synthetic evaluation data, corpus

15.8.2 Robustness and security

This subsection explains how the retrieval corpus creates security and privacy risks that current defenses do not fully address.

Key terms: retrieval corpus, prompt injection, poisoned document, content filtering, obfuscated, data privacy, access control, trust mechanisms

15.8.3 The efficiency-quality tradeoff

This subsection explains why quality-enhancing RAG techniques must be weighed against production latency budgets and why there is no universal way to choose the best configuration.

Key terms: latency, reranking, multi-step retrieval, agentic RAG, Self-RAG, context compression, latency budgets, RAGGED, stability and scalability metrics, quality-latency frontier

15.8.4 Retrieval across heterogeneous sources

This section explains why retrieving answers across disconnected enterprise sources remains difficult even when individual documents can be parsed and indexed.

Key terms: enterprise knowledge, indexes, routing layer, Graph RAG, knowledge graph, modalities, entity relationships

15.8.5 Open questions

This section identifies unresolved RAG limitations, sketches research directions for addressing them, and recommends testing new techniques on your own workload.

Key terms: continuous pre-training with retrieval, differentiable retrieval pipelines, end-to-end gradient-based optimization, federated RAG, privacy-preserving retrieval, federated search protocols, aggregation and arithmetic, top-5 retrieval, SQL query, analytics API

15.9 Summary

Concepts explained in this chapter

← Chapter 14: Evaluating RAG systems

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.