Chapter 11: Correction, planning, and reasoning
This introduction explains why better retrieval alone cannot fix single-pass RAG and contrasts two ways to add control: correcting retrieval afterward and planning it beforehand.
After this chapter you will be able to
- Single-pass RAG weakness. Explain why improved indexing does not eliminate the risk of wrong answers in a single-pass Retrieve-then-Generate pipeline, and why the system needs control over what happens after retrieval.
- CRAG correction-after approach. Describe CRAG's correction-after approach: audit retrieved documents, take a fallback action when evidence is weak or stale, and filter noisy context before generation.
- LevelRAG planning-first approach. Describe how LevelRAG plans retrieval for a multi-part question, routes different sub-queries to suitable searchers, and checks whether the combined evidence answers the original question.
- Agentic RAG control flow. Explain how Agentic RAG differs from Single-Pass RAG: it can decide when and what to retrieve, assess results, adapt, and repeat before answering.
- PEP onboarding answer requirements. Explain why the expedited-KYC question cannot be answered safely from a single apparent match, including what must be checked and what a wrong answer could cost.
- Similarity versus document validity. Explain why the single-pass retriever ranked an outdated internal policy above the restrictive EU directive, and why that ranking was unsafe.
- Snapshot knowledge and freshness gap. Explain why Single-Pass RAG may give an outdated answer to a question about current regulatory status, even when it narrows retrieval effectively.
- Freshness checks and search fallback. Describe the corrective response proposed here when cached evidence may be stale or insufficient.
- Long-tail retrieval collapse. Explain why a dense retriever can rank a famous namesake above the obscure entity a question actually asks about.
- Multi-hop comparison requirements. Describe why the beneficial-ownership question requires facts from more than one source: find the UK threshold, find the EU threshold, and compare them.
- Single-pass retrieval failure. Explain how retrieving once with the entire comparative query can leave out one jurisdiction's evidence and lead the model to invent the missing part of an answer.
- Fixed versus adaptive decomposition. Distinguish fixed, parallel query expansion from an agentic approach that plans subqueries around the question, checks whether each result is sufficient, and searches again if needed.
- Retrieved document quality gate. Explain that CRAG evaluates retrieved documents between retrieval and generation, then adapts if the sources are insufficient, potentially seeking better sources on the Web.
- Retrieval Evaluator role. Explain that CRAG's Retrieval Evaluator estimates how relevant a retrieved document is to a query, rather than synthesizing the final answer.
- Three confidence-zone actions. Trace what CRAG does when the evaluator judges retrieved documents Correct, Incorrect, or Ambiguous.
- Decompose-then-Recompose process. Trace how CRAG splits a retrieved document into knowledge strips, filters them against the query, and recombines the useful strips into context for the generator.
- Web search fallback path. Explain when CRAG falls back to web search and how it prepares and refines the search results.
- Evaluator and pipeline responsibilities. Describe how RetrievalEvaluator scores and refines documents while CRAGPipeline orchestrates retrieval, evaluation, and the choice between internal evidence and external search.
- Quality gate and fallback. Explain how evaluator thresholds select the internal-evidence or web-search branch, and why an external fallback must be aimed at trustworthy sources.
- Compliance trace failure. Trace why the compliance example enters the ambiguous branch yet still gives the generator the stale permission rather than the correction.
- Evaluator calibration limits. Explain why a borrowed relevance scorer and reused thresholds fail in this example, and describe the two remedies the section considers.
- CRAG performance and latency. Explain how CRAG performed against standard RAG and what costs accompany its gains.
- Value of external evidence checks. Explain why a separate evidence-evaluation step can help even when a generator reflects on retrieval or reasons at inference time.
- Single-correction planning limit. Distinguish CRAG’s ability to correct a retrieval from the ability to plan retrievals for a multi-part question.
- Hierarchical search responsibilities. Explain how LevelRAG separates deciding what to search for from carrying out each search: the High-Level Searcher plans and routes atomic sub-queries, while specialized Low-Level Searchers retrieve evidence.
- Evidence verification feedback loop. Trace LevelRAG’s stated loop from decomposition and search through evidence summaries and verification, including what happens when the combined evidence does not yet answer the original question.
- High-Level Searcher Planning. Explain how the High-Level Searcher plans a complex request as atomic information needs without searching database indices itself, and how retrieved evidence can prompt a new query.
- Query-Focused Evidence Summaries. Describe how the HLS condenses returned documents by answering each atomic query, and why this is preferable to concatenating documents or making plain summaries.
- Verify-and-Supplement Loop. Trace how the HLS checks the combined summaries against the original request, searches for missing information, and decides when to stop.
- Specialized searcher roles. Distinguish when LevelRAG uses sparse, dense, and web searchers, and explain why combining them helps with both exact entities and broad topics.
- Adaptive retrieval feedback loop. Trace how the high-level searcher responds to overly broad or irrelevant results and explain why that makes LevelRAG agentic.
- Plan and verify loop. Trace how the HighLevelSearcher and LevelRAGEngine decompose a question, retrieve and summarize evidence, verify whether it answers the question, and supplement gaps until evidence is sufficient or an iteration limit is reached.
- Compliance conflict and evaluator. Explain what the compliance example shows about decomposition and why LevelRAG alone cannot resolve conflicting policies.
- Reactive and proactive agency. Distinguish where CRAG and LevelRAG place agency in a RAG pipeline: CRAG audits retrieval afterward, while LevelRAG structures the query before retrieval and verifies coverage.
- Selection tradeoffs and failure modes. Justify choosing LevelRAG for complex multi-hop coverage or CRAG for simpler fact-checking at risk of staleness, accounting for their latency and integration costs.
- Composite agent sequence. Trace how the section combines LevelRAG and CRAG: plan by decomposing the query, execute retrieval for each atomic sub-query, and correct a low-confidence sub-query with a targeted fallback.
- Agentic control flow spectrum. Distinguish four levels of agentic RAG by how much control the system has: a single evaluation and fallback, an initial plan without feedback, a plan-evaluate-replan loop, and retrieval combined with arbitrary tools.
- Cost guided agentic adoption. Justify starting with a measured single-pass RAG pipeline and adding agentic control only when its observed failures and the cost of wrong answers warrant the extra latency and tokens.
- Agentic RAG product examples. Describe how the named products reflect LevelRAG-style planning or decomposition and CRAG-style evidence checking.
- Per Subquery Planning and Correction. Trace how a LevelRAG-style searcher breaks a query into sub-queries and how a CRAG-style evaluator checks and corrects each result before the combined summaries are verified.
- Confidence-Based Answer Refusal. Explain why a system may refuse an answer or escalate it to a human when the supporting retrieval has a low confidence score, particularly in regulated industries.
What it builds on
- From lists to trees (chapter 10) · RAPTOR's tree organization is an example of improving the index rather than the pipeline's control flow.
- Naive RAG: Establishing the foundation (chapter 1) · The basic single-pass retrieval-and-answer pattern provides context for the contrast drawn here.
- Implementing multi-query RAG systems (chapter 8) · Multi-query retrieval that combines results from expanded queries.
- Reflection tokens for retrieval, relevance, and critique (chapter 9) · Self-RAG’s reflection tokens
- The retriever (chapter 4) · Introduces retrieval as a component of RAG.
- Generating hypothetical documents (chapter 7) · Introduces generating hypothetical documents for retrieval.
Section by section
Introduction
This introduction explains why better retrieval alone cannot fix single-pass RAG and contrasts two ways to add control: correcting retrieval afterward and planning it beforehand.
- Single-pass RAG weakness: Explain why improved indexing does not eliminate the risk of wrong answers in a single-pass Retrieve-then-Generate pipeline, and why the system needs control over what happens after retrieval.
Why it matters: A better index cannot protect a high-stakes answer if the system never checks whether the retrieved evidence fits the question. - CRAG correction-after approach: Describe CRAG's correction-after approach: audit retrieved documents, take a fallback action when evidence is weak or stale, and filter noisy context before generation.
Why it matters: Checking and correcting retrieval helps keep irrelevant or stale evidence from becoming a confident answer. - LevelRAG planning-first approach: Describe how LevelRAG plans retrieval for a multi-part question, routes different sub-queries to suitable searchers, and checks whether the combined evidence answers the original question.
Why it matters: Planning and checking coverage give a multi-part question a better chance of receiving evidence for all its parts.
Key terms: Retrieve-then-Generate, single-pass, control flow, CRAG, correction-after, Retrieval Evaluator, fallback, decompose-then-recompose, LevelRAG, plan-before
11.1 The agentic turn
This section defines the shift from single-pass RAG to agentic RAG and identifies recurring retrieval and answer failures that motivate it.
- Agentic RAG control flow: Explain how Agentic RAG differs from Single-Pass RAG: it can decide when and what to retrieve, assess results, adapt, and repeat before answering.
Why it matters: This distinction explains why changing control flow can address failures that better indexing alone cannot. - Recurring RAG failure points: Distinguish the four production failure points listed here: missing or stale content, a relevant document ranked too low, a generic document substituted for a specific one, and an answer that omits part of a multi-part question.
Why it matters: Naming the failure precisely helps distinguish what went wrong before considering a remedy.
Key terms: Agentic RAG, Single-Pass RAG, control flow, cyclical feedback loop, validation step, FP1: Missing Content or Staleness, FP2: Missed the Top Rank, FP6: Incorrect Specificity, FP7: Incomplete Answers
11.1.1 The enterprise compliance assistant
The section introduces a bank compliance scenario to show why a high-stakes onboarding question requires checking several potentially conflicting sources rather than relying on a single-pass RAG answer.
- Heterogeneous compliance knowledge base: Describe the kinds of information the Enterprise Compliance Assistant must work with and why they may be difficult to reconcile.
Why it matters: The variety and potential conflict among sources make this more than a simple document lookup. - PEP onboarding answer requirements: Explain why the expedited-KYC question cannot be answered safely from a single apparent match, including what must be checked and what a wrong answer could cost.
Why it matters: An answer that overlooks an approval condition could expose the bank to major regulatory penalties.
Key terms: Enterprise Compliance Assistant, Internal Policy Documents, External Regulatory Directives, Sanctions Lists, Client Profiles, politically exposed person (PEP), expedited KYC, Single-Pass RAG, approval matrices
11.1.2 Anatomy of a single-pass failure
This section uses a retrieval trace to show how a single-pass system can favor a similar but invalid policy, then identifies the checks that could catch the error.
- Similarity versus document validity: Explain why the single-pass retriever ranked an outdated internal policy above the restrictive EU directive, and why that ranking was unsafe.
Why it matters: A highly similar retrieval result can still be the wrong evidence for an answer. - Proposed correction and search: Describe the two corrective responses named for the date discrepancy and the missing EU rule.
Why it matters: The proposed checks target the validity problem that similarity ranking missed.
Key terms: Single-Pass system, vector similarity score, validity, metadata dates, EU Directive, Internal Policy, CRAG evaluator, Ambiguous, LevelRAG planner, EU PEP Rules 2024
11.1.3 Stale knowledge and missing freshness checks
This section explains why Single-Pass RAG can return outdated regulatory evidence and what a freshness check and live-search fallback would add.
- Snapshot knowledge and freshness gap: Explain why Single-Pass RAG may give an outdated answer to a question about current regulatory status, even when it narrows retrieval effectively.
Why it matters: A relevant-looking document can still give the wrong answer to a question about what is true now. - Freshness checks and search fallback: Describe the corrective response proposed here when cached evidence may be stale or insufficient.
Why it matters: A freshness check gives the system a way to notice when its cached evidence may not support a current answer.
Key terms: Single-Pass RAG, snapshot of knowledge, staleness, FP1: Missing Content or Staleness, evaluator, potentially stale, live web search, cached evidence
11.1.4 Retrieval collapse on long-tail entities
This section explains how retrieval can favor a famous namesake over an obscure entity and why that failure is especially serious in compliance work.
- Long-tail retrieval collapse: Explain why a dense retriever can rank a famous namesake above the obscure entity a question actually asks about.
Why it matters: A plausible top result may belong to the wrong person or company. - Compliance impact and failure types: Describe why confusing similarly named people matters in Know Your Customer work, and distinguish the two failure types the section assigns to that error.
Why it matters: The distinction shows why a seemingly plausible search result can create a real compliance risk.
Key terms: long-tail entities, dense retriever, Retrieval Collapse, Know Your Customer (KYC), FP2: Missed the Top Rank, FP6: Incorrect Specificity
11.1.5 Multi-hop reasoning gaps
This section explains why comparative questions can defeat single-pass RAG and how adaptive retrieval addresses the gaps left by fixed query decomposition.
- Multi-hop comparison requirements: Describe why the beneficial-ownership question requires facts from more than one source: find the UK threshold, find the EU threshold, and compare them.
Why it matters: Recognizing the separate facts needed for a comparison helps reveal when retrieved evidence is incomplete. - Single-pass retrieval failure: Explain how retrieving once with the entire comparative query can leave out one jurisdiction's evidence and lead the model to invent the missing part of an answer.
Why it matters: A plausible-sounding comparison is unreliable if retrieval supplied evidence for only one side. - Fixed versus adaptive decomposition: Distinguish fixed, parallel query expansion from an agentic approach that plans subqueries around the question, checks whether each result is sufficient, and searches again if needed.
Why it matters: The distinction explains why splitting a query alone may not resolve missing evidence.
Key terms: multi-hop reasoning, comparative, UK threshold, EU threshold, Single-Pass RAG, fragmented context, FP7: Incomplete Answers, RAG-Fusion, multi-query retrieval, HyDE
11.2 Corrective retrieval augmented generation
The section introduces CRAG as a lightweight way for a RAG system to check retrieved documents and seek better sources before generating an answer.
- Retrieved document quality gate: Explain that CRAG evaluates retrieved documents between retrieval and generation, then adapts if the sources are insufficient, potentially seeking better sources on the Web.
Why it matters: The quality check lets the system respond to weak retrieval rather than immediately writing from it. - Lightweight pipeline modification: Describe why CRAG can be added to an existing RAG pipeline without changing its initial retrieval or index layer.
Why it matters: This distinction explains why an existing pipeline can adopt CRAG without rebuilding its retrieval infrastructure.
Key terms: Corrective Retrieval Augmented Generation (CRAG), Trust, but Verify, quality check, fallback, evaluation node, agentic spectrum, index layer
11.2.1 The retrieval evaluator
This section explains how CRAG's Retrieval Evaluator scores retrieved evidence, routes the system to corrective actions, and remains limited by documents the retriever failed to return.
- Retrieval Evaluator role: Explain that CRAG's Retrieval Evaluator estimates how relevant a retrieved document is to a query, rather than synthesizing the final answer.
Why it matters: Separating relevance judgment from answer writing explains how CRAG can check retrieval before generating an answer. - Three confidence-zone actions: Trace what CRAG does when the evaluator judges retrieved documents Correct, Incorrect, or Ambiguous.
Why it matters: The three-way decision prevents CRAG from treating questionable retrieval as automatically ready for generation. - Retrieval-set blindness limit: Explain why a Correct verdict cannot guarantee that the best or newest evidence was found, and what limited escape route the web-search branch offers.
Why it matters: A convincing relevance score is not proof that retrieval surfaced the evidence an answer most needs.
Key terms: Retrieval Evaluator, discriminative task, scalar confidence score, generative LLM, Correct, Incorrect, Ambiguous, Knowledge Refinement, Web Search, Hybrid strategy
11.2.2 Decompose-then-recompose
This section explains how CRAG turns a retrieved document into a shorter, more relevant context for the generator.
- Decompose-then-Recompose process: Trace how CRAG splits a retrieved document into knowledge strips, filters them against the query, and recombines the useful strips into context for the generator.
Why it matters: This is how CRAG keeps useful evidence while removing irrelevant parts of a retrieved document. - Document noise and focused context: Explain why passing an entire relevant document to the generator can be harmful and how selecting only the needed material helps.
Why it matters: Finding the right document is not enough if the answer is buried in unrelated text.
Key terms: Decompose-then-Recompose, knowledge strips, Evaluator, recomposition, Lost-in-the-Middle errors, signal-to-noise ratio, boilerplate
11.2.3 Corrective actions and web search integration
This section explains how CRAG uses web search when internal retrieval is inadequate and combines external findings with internal context.
- Web search fallback path: Explain when CRAG falls back to web search and how it prepares and refines the search results.
Why it matters: The fallback gives CRAG a way to seek information that its internal corpus may not contain. - Internal and external evidence: Describe how the sanctions example combines an outdated internal record with a current external finding.
Why it matters: The example shows why a current web finding can complement, rather than simply replace, an internal record.
Key terms: Incorrect, Ambiguous, web search fallback, query rewriter, Decompose-then-Recompose, internal context, external finding, OFAC
11.2.4 The CRAG pipeline
This section implements a branching CRAG pipeline for a compliance query and shows how evaluator and threshold choices can either catch stale evidence or filter out the correction.
- Evaluator and pipeline responsibilities: Describe how RetrievalEvaluator scores and refines documents while CRAGPipeline orchestrates retrieval, evaluation, and the choice between internal evidence and external search.
Why it matters: The distinction shows where to change evidence-quality judgments without rewriting the workflow. - Quality gate and fallback: Explain how evaluator thresholds select the internal-evidence or web-search branch, and why an external fallback must be aimed at trustworthy sources.
Why it matters: A corrective search is useful only if the branch fires appropriately and the outside evidence is credible. - Compliance trace failure: Trace why the compliance example enters the ambiguous branch yet still gives the generator the stale permission rather than the correction.
Why it matters: Taking the corrective branch does not help if the later filter removes the evidence needed to correct the answer. - Evaluator calibration limits: Explain why a borrowed relevance scorer and reused thresholds fail in this example, and describe the two remedies the section considers.
Why it matters: An uncalibrated evaluator can turn a stale policy into a confident answer without producing an obvious error.
Key terms: RetrievalEvaluator, CRAGPipeline, branching control flow, quality gate, ambiguity zone, Incorrect, Ambiguous, web_searcher, Decompose-then-Recompose, cross-encoder
11.2.5 Evaluation, shortcomings, and where CRAG fits
This section evaluates CRAG’s benefits and costs, shows why external evidence checks remain useful, and defines the limits of its defensive-retrieval pattern.
- CRAG performance and latency: Explain how CRAG performed against standard RAG and what costs accompany its gains.
Why it matters: The accuracy benefit has to be weighed against the time spent on corrective search. - Value of external evidence checks: Explain why a separate evidence-evaluation step can help even when a generator reflects on retrieval or reasons at inference time.
Why it matters: A model’s ability to reason or critique itself does not eliminate the value of checking what evidence reaches it. - Single-correction planning limit: Distinguish CRAG’s ability to correct a retrieval from the ability to plan retrievals for a multi-part question.
Why it matters: Auditing one retrieval does not solve a question that requires a plan for several retrievals. - Defensive retrieval beyond CRAG: Describe CRAG’s lasting design pattern and how it appears in tooling even when the CRAG name is absent.
Why it matters: The useful idea is the pre-generation evidence check, not whether a product uses the paper’s name or exact fallback.
Key terms: CRAG, web search fallback, retrieval noise, latency trade-offs, Self-CRAG, external evaluator, reflection tokens, Search-o1, Reason-in-Documents, agentic spectrum
11.3 LevelRAG
Introduces LevelRAG as a hierarchical way to plan and verify retrieval, and distinguishes its approach from CRAG’s correction of retrieved results.
- Hierarchical search responsibilities: Explain how LevelRAG separates deciding what to search for from carrying out each search: the High-Level Searcher plans and routes atomic sub-queries, while specialized Low-Level Searchers retrieve evidence.
Why it matters: This separation lets the system choose a retrieval method for each part of a complex question. - Evidence verification feedback loop: Trace LevelRAG’s stated loop from decomposition and search through evidence summaries and verification, including what happens when the combined evidence does not yet answer the original question.
Why it matters: Retrieving evidence for every initial sub-query does not by itself show that the original question has been answered. - Planning and correction approaches: Distinguish LevelRAG’s planning of retrieval from CRAG’s correction of retrieved results, and describe how LevelRAG extends earlier approaches that decompose questions or alternate reasoning with retrieval.
Why it matters: The distinction clarifies why planning and correction can be combined without treating CRAG as part of LevelRAG.
Key terms: High-Level Searcher, Low-Level Searchers, atomic sub-queries, adaptive routing, feedback loop, verify, supplemental queries, CRAG, LevelRAG, IRCoT
11.3.1 The high-level searcher
The section explains how LevelRAG’s High-Level Searcher plans research, condenses retrieved evidence, and checks whether it is sufficient to answer a complex question.
- High-Level Searcher Planning: Explain how the High-Level Searcher plans a complex request as atomic information needs without searching database indices itself, and how retrieved evidence can prompt a new query.
Why it matters: A complex answer may require both separate lines of research and follow-up questions revealed by the evidence. - Query-Focused Evidence Summaries: Describe how the HLS condenses returned documents by answering each atomic query, and why this is preferable to concatenating documents or making plain summaries.
Why it matters: Compact, question-focused evidence helps the system retain relevant facts without overloading its context. - Verify-and-Supplement Loop: Trace how the HLS checks the combined summaries against the original request, searches for missing information, and decides when to stop.
Why it matters: Checking for missing evidence prevents the system from treating its first retrieval pass as a complete answer.
Key terms: High-Level Searcher, HLS, Multi-hop Logic Planning, Query Decomposition, atomic queries, per-query summarization, atomic query, context window, Lost-in-the-Middle, Verify
11.3.2 The low-level searcher
This section explains how LevelRAG combines specialized searchers and uses retrieval results to decide what to search next.
- Specialized searcher roles: Distinguish when LevelRAG uses sparse, dense, and web searchers, and explain why combining them helps with both exact entities and broad topics.
Why it matters: Choosing a search method suited to the query gives LevelRAG more ways to find useful evidence. - Search query refinement methods: Describe how sparse search refines an atomic query and how dense search uses a generated passage to improve conceptual retrieval.
Why it matters: Each searcher can reshape a query to make its own retrieval method more effective. - Adaptive retrieval feedback loop: Trace how the high-level searcher responds to overly broad or irrelevant results and explain why that makes LevelRAG agentic.
Why it matters: Feedback lets LevelRAG correct a poor search rather than treating its first results as final.
Key terms: Low-Level Searchers, Sparse searcher, Dense searcher, Web searcher, Lucene syntax operators, Extend, Filter, Emphasize, Pseudo-Document Generation, atomic query
11.3.3 LevelRAG implementation
This section implements LevelRAG’s plan-and-verify retrieval loop, then uses a compliance example and evaluation findings to show its benefits, limits, and operational costs.
- Plan and verify loop: Trace how the HighLevelSearcher and LevelRAGEngine decompose a question, retrieve and summarize evidence, verify whether it answers the question, and supplement gaps until evidence is sufficient or an iteration limit is reached.
Why it matters: Verification makes retrieval responsive to missing evidence rather than ending after the first batch of searches. - Low-level searcher routing: Distinguish the implementation’s single-searcher routing from the paper’s all-searcher approach, including the cost and flexibility tradeoff.
Why it matters: The routing choice affects both deployment cost and whether this implementation reproduces the paper’s design. - Compliance conflict and evaluator: Explain what the compliance example shows about decomposition and why LevelRAG alone cannot resolve conflicting policies.
Why it matters: Finding relevant evidence is not the same as deciding which of two conflicting sources to trust. - Planning cost and variability: Describe why LevelRAG’s loop can be expensive and why downstream systems should not assume identical plans across runs.
Why it matters: A useful retrieval plan still needs a cost budget and downstream logic that tolerates changing plans. - Structured retrieval findings: State the reported benefits of explicit decomposition and heterogeneous search for multi-hop, precise fact, and conceptual queries.
Why it matters: Different query types benefit from different retrieval tactics rather than one undifferentiated vector search.
Key terms: HighLevelSearcher, LevelRAGEngine, atomic query, verify, supplement, Low-Level Searchers, single-searcher routing, retrieval modality, decomposition, INSUFFICIENT
11.4 Reactive versus proactive agency
The section contrasts CRAG’s reactive correction with LevelRAG’s proactive planning, compares when each is useful, and shows how they can work together.
- Reactive and proactive agency: Distinguish where CRAG and LevelRAG place agency in a RAG pipeline: CRAG audits retrieval afterward, while LevelRAG structures the query before retrieval and verifies coverage.
Why it matters: The point at which a system intervenes determines which retrieval failures it can address. - Selection tradeoffs and failure modes: Justify choosing LevelRAG for complex multi-hop coverage or CRAG for simpler fact-checking at risk of staleness, accounting for their latency and integration costs.
Why it matters: Choosing a pattern means balancing the failure you need to fix against response time and migration effort. - Composite agent sequence: Trace how the section combines LevelRAG and CRAG: plan by decomposing the query, execute retrieval for each atomic sub-query, and correct a low-confidence sub-query with a targeted fallback.
Why it matters: The combination preserves planning for complex questions while correcting only the retrievals that need it.
Key terms: CRAG, LevelRAG, reactive, proactive, atomic sub-queries, verify step, staleness, multi-hop, latency, integration
11.5 Agentic RAG in practice
Introduces a broader view of agentic RAG, previewing how CRAG and LevelRAG relate to deployed systems.
11.5.1 The agentic spectrum
This section places agentic RAG approaches on a spectrum of control flow and explains when their added cost is justified.
- Agentic control flow spectrum: Distinguish four levels of agentic RAG by how much control the system has: a single evaluation and fallback, an initial plan without feedback, a plan-evaluate-replan loop, and retrieval combined with arbitrary tools.
Why it matters: The spectrum lets you compare agentic systems by what they actually control rather than calling them simply agentic or not. - Cost guided agentic adoption: Justify starting with a measured single-pass RAG pipeline and adding agentic control only when its observed failures and the cost of wrong answers warrant the extra latency and tokens.
Why it matters: It helps you spend on more elaborate retrieval only where the resulting improvement is worth its operational cost.
Key terms: lightly agentic, plan-then-execute, plan-evaluate-replan, tool-using research agents, CRAG, LevelRAG, single pass, latency, tokens, regulated industry
11.5.2 Products and platforms
This subsection connects agentic RAG ideas to products and frameworks that support planning, evidence checking, and citation grounding.
- Agentic RAG product examples: Describe how the named products reflect LevelRAG-style planning or decomposition and CRAG-style evidence checking.
Why it matters: These examples show how planning and evidence checks appear in agentic RAG products. - Framework agent primitives: State what the named frameworks make accessible for building agentic RAG systems.
Why it matters: The frameworks provide building blocks for implementing the planning and evaluation roles described in the chapter.
Key terms: agentic RAG, query decomposition, retrieval strategies, evidence audits, citation grounding, LangGraph, Haystack, LlamaIndex Workflows, agent primitives, ReAct (Reason-Act)
11.5.3 How CRAG and LevelRAG fit together in production
Shows how a production pipeline combines LevelRAG planning with CRAG correction, then limits its compute cost by routing simple queries elsewhere.
- Per Subquery Planning and Correction: Trace how a LevelRAG-style searcher breaks a query into sub-queries and how a CRAG-style evaluator checks and corrects each result before the combined summaries are verified.
Why it matters: This explains how planning and correction work together without treating every retrieval result as equally reliable. - Query Complexity Routing: Justify why production deployments gate the combined agentic loop on query complexity and describe where simple queries go.
Why it matters: Routing avoids paying for the full agentic loop when a simpler path is sufficient.
Key terms: HighLevelSearcher, Low-Level Searcher, atomic sub-queries, CRAG-style evaluator, verify step, query complexity, classifier, agentic pipeline, single-pass path
11.5.4 Refusal as a feature
The section presents refusal based on retrieval confidence as a feature of corrective RAG and contrasts one-shot question answering with planned research by agents.
- Confidence-Based Answer Refusal: Explain why a system may refuse an answer or escalate it to a human when the supporting retrieval has a low confidence score, particularly in regulated industries.
Why it matters: An assistant should not present an answer as reliable when its retrieved support is weak. - Planned Research Versus One-Shot Answers: Describe the shift associated with LevelRAG’s query planning from one-shot question-answering bots toward research agents that take longer but reason more deeply.
Why it matters: A slower answer can be an intentional tradeoff when a task calls for deeper research.
Key terms: Retrieval Validation, Confidence Score, Refusal as a Feature, LevelRAG, Query Planning, one-shot QA bots, Research Agents
11.6 Summary
Where this is used later
- The reader is the bottleneck (chapter 15) · Introduces corrective actions for low-quality retrieval.
- Mapping scores to failure points (chapter 14) · Connects observed answer failures to evaluator and retrieval-stage choices.
- When agentic RAG helps and when it does not (chapter 15) · Informs when the cost of agentic RAG is justified.
- The evolving RAG landscape (chapter 15) · Introduces degrees of agency in RAG systems.
- Mapping failure modes to RAG approaches (chapter 15) · Supports matching observed failure modes to RAG approaches.
- From patterns to pipelines (chapter 15) · Provides a concrete example of combining agentic patterns into a pipeline.
Concepts explained in this chapter
- CRAGThe Comprehensive RAG Benchmark is a benchmark for evaluating retrieval-augmented generation systems on question-answering performance.
- LevelRAGLevelRAG is a retrieval approach that decomposes a complex question into atomic queries, routes them to searchers, summarizes results, and iterates until verification passes.
- retrieval evaluatorA retrieval evaluator is a component that assesses retrieval results and can trigger searches for additional supporting sources.
- atomic queryA sub-query is an individual query produced by dividing a larger information need into smaller retrieval or reasoning steps.
- High-Level SearcherThe High-Level Searcher is LevelRAG's planning component that decomposes a user query into atomic sub-queries, summarizes evidence, verifies answers, and issues supplemental queries.
- retrieve-then-generate pipelineSingle-pass RAG is a retrieval-augmented generation architecture that performs one retrieval pass and then generates an answer without iterative planning or correction.
- decompose-then-recomposeDecompose-then-recompose is a context-processing technique that splits retrieved documents into fine-grained segments, filters them, and recombines the useful text.
- agentic RAGAgentic RAG is a retrieval-augmented generation architecture in which runtime decisions control retrieval, reasoning, or tool actions based on system state and signals.
- routingA query router is a system component that analyzes a query and directs it to an appropriate retrieval or processing path.
- crag pipelineA CRAG pipeline is an orchestration workflow that retrieves documents, evaluates them, and branches to refinement or corrective retrieval.
- proactive planningProactive planning is a retrieval strategy that structures an information need and plans retrieval steps before initial retrieval.
- continuous improvementA closed-loop system observes an intermediate or resulting state and uses that evaluation to decide subsequent actions.
- verificationVerification is the structured assessment of whether gathered evidence sufficiently answers a question.
- query decompositionQuery decomposition is the technique of splitting a complex information need into smaller sub-queries that can be searched or answered separately.
- confidenceA confidence score is a numerical indication of a system's estimated certainty in an answer or prediction.
- lexical searchLexical search retrieves text by matching words or other surface forms rather than representing meaning directly.
- stale dataStaleness is the condition in which stored or indexed information no longer reflects the current underlying source.
- active correctionReactive correction is a retrieval pattern that audits an initial retrieval result and intervenes only when that result is inadequate.
- HyDEHyDE (Hypothetical Document Embeddings) is a retrieval technique that generates a hypothetical document or answer and embeds it to improve document retrieval.
- similarity searchSimilarity search finds items whose representations are close to a query representation according to a similarity or distance function.
- embeddingA dense vector is a vector whose coordinates are generally populated with informative numerical values.
- query rewritingQuery reformulation is the transformation of an existing search query into an alternative formulation intended to retrieve more suitable information.
- GraphRAGGraph RAG is a retrieval-augmented generation architecture that represents corpus relationships as a graph and retrieves evidence through graph structure.
- cross-encoderA cross-encoder jointly processes two input sequences through one encoder so that tokens from both sequences can interact directly during encoding.
- model calibrationCalibration is the degree to which a model's confidence or categorical assessments agree with externally observed outcomes or labels.
- self-correctionSelf-monitoring and correction is a system capability that evaluates outputs and information quality and initiates corrective actions when problems are detected.
- text chunkA chunk is a smaller contiguous segment of source text created for processing or retrieval.
- rerankingA reranker is a model or retrieval component that reorders candidate results according to estimated relevance.
- vector databaseA vector database is a database system designed to store and retrieve vector representations using similarity search.
- retrieval validationRetrieval validation is the assessment of retrieved evidence to determine whether it is sufficiently reliable for answering.
- groundingGrounding is conditioning a generated answer on retrieved evidence so that its claims can be checked against supporting passages.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
