Chapter 1: How RAG research prevents disasters
The introduction defines RAG, shows why access to a correct source does not guarantee a grounded answer, and frames reliability as an engineering problem.
After this chapter you will be able to
- RAG and other answer systems. Distinguish RAG from search engines and standalone language models: it retrieves external information and uses that evidence to generate an answer.
- Source access versus grounding. Explain what the Air Canada chatbot case illustrates about the difference between having the correct document nearby and giving an answer grounded in it, without claiming the chatbot used retrieval.
- Failure-focused research literacy. Explain how recognizing persistent architectural patterns and production failure modes helps teams choose research-backed fixes instead of guessing.
- Measurement before added complexity. Justify starting with the simplest pipeline that meets an accuracy target and adding components only in response to measured failures.
- Retrieval and generation together. Explain why RAG combines retrieval with language-model generation rather than relying on either alone. Describe how external sources can make a synthesized answer current, grounded, and attributable.
- Offline knowledge base preparation. Describe what the indexing pipeline does before a user asks a question and distinguish it from the real-time query pipeline.
- Real-time query sequence. Trace the four stages of the query pipeline and state the purpose of each stage.
- Three production AI pitfalls. Explain the knowledge boundary, hallucination challenge, and private knowledge gap, and relate them to the risks of deploying a generic chatbot.
- Training cutoff and knowledge boundary. Explain that an LLM’s knowledge boundary is where its training-based world knowledge ends, so it cannot know the outcome of an event that happened after training.
- External retrieval for current knowledge. Explain why an AI assistant must actively retrieve changing organizational knowledge rather than rely only on its trained parameters, and how access to current information can improve decisions.
- Plausible but ungrounded answers. Explain what a hallucination is, why it can be difficult to notice, and why a larger model does not necessarily solve that problem.
- Types of hallucination. Distinguish the hallucination types named in this section by what goes wrong in each case.
- Evidence checking and its limits. Distinguish spotting a suspicious answer without evidence from checking it against retrieved evidence, and state the limit of those checks.
- Private knowledge access gap. Explain why a model trained on public internet data cannot, by itself, answer questions that depend on an organization’s private or specialized information.
- Changing knowledge and reuse. Distinguish fine-tuning on historical logs from using RAG when policies change, and describe how the same RAG workflows can serve different knowledge bases.
- Granular private data controls. Explain why access to private organizational data requires controls outside model weights, including restrictions on who can access which information.
- RAG's enduring role. Explain why RAG remains an architectural pattern despite advances in other techniques. Describe its core function as connecting generative models to external data and providing the right context for a task.
- Retrieval versus longer context. Distinguish the benefit of longer context windows from the benefit of retrieving relevant information first. Explain why putting a large collection into a prompt can be less effective than selecting the pertinent parts.
- RAG as context engineering. Describe how RAG constructs useful context for a model: it selects relevant chunks, puts them in the prompt, and filters out irrelevant information rather than changing the model's weights.
- Changing and rare knowledge. Explain why retrieval is useful when facts change faster than a model can be retrained or are too rare to have been learned reliably.
- Grounding and source support. Distinguish using a passage to improve an answer from returning it so a person can check the answer, and explain why a citation alone does not establish support.
- Retrieval costs and synthesis value. Justify when retrieval can be economical and why RAG can deliver value beyond keyword search despite added infrastructure costs.
- Three layers of RAG failure. Distinguish retrieval, augmentation, and generation failures using the examples given in the section.
- Failure taxonomy for diagnosis. Explain how a catalog of specific failure points makes production RAG problems easier to diagnose and prioritize.
- Ranking And Answer Specificity. Trace the loan-officer example from the buried current memo to the generic answer, distinguishing the retrieval-ranking failure from the answer-specificity failure.
- Fabrication Despite Relevant Documents. Explain why retrieved documents do not eliminate hallucinations, using the fabricated compliance advice as an example.
- Silent Answer Failure Points. Distinguish missing content, content cut from context, overlooked text, and incomplete answers as ways a RAG response can fail without an obvious error.
- Measurable RAG Reliability. Describe the section's definition of reliability, distinguishing an observed failure from its underlying fault and explaining why provenance alone cannot detect silent degradation.
- Failure-driven architectural research. Explain why the failures described here motivate changes to RAG architecture rather than prompt wording or abandonment of RAG.
- Modular and agentic boundary. Distinguish modular from agentic RAG by who determines the possible steps, and explain the resulting difference in cost predictability.
- Naive RAG pipeline. Trace how naive RAG turns documents and a query into a generated answer.
- Retrieval limits and mitigations. Explain how naive RAG can miss needed context and state the limitations and mitigations identified here.
- Advanced RAG architecture. Explain how advanced RAG relates to the basic RAG pipeline and what kinds of improvement it pursues.
- Document and query preparation. Describe how preprocessing and query enhancements improve what is available to retrieval and how questions are sent to it.
- Retrieval quality refinements. Describe the retrieval improvements listed here and how the section’s multi-query example combines them.
- Retrieved context optimization. Explain why advanced RAG compresses, deduplicates, and orders retrieved context for the generator.
- Interchangeable specialized components. Explain how modular RAG differs from a fixed pipeline by combining interchangeable components specialized for retrieval methods, content types, or tasks.
- Query-specific module routing. Describe how a modular system can arrange and select combinations of modules at query time according to the query or performance requirements.
- Autonomous retrieval and timing. Explain how agentic RAG differs from a pipeline that retrieves once before generation, and why a reasoning model may need evidence later.
- Adaptive information gathering capabilities. Describe how agentic RAG can formulate queries, retrieve repeatedly, assess its output, and coordinate information gathering for complex tasks. Distinguish its autonomous control from critique or validation steps, which other RAG pipelines may also have.
- Bounded autonomy tradeoffs. Justify when agentic RAG is useful and why teams often limit its autonomy. Describe the costs and predictability concerns and the kinds of controls the section gives.
- Business-driven RAG architecture choice. Distinguish when the section recommends naive, advanced, modular, or agentic RAG based on constraints such as retrieval quality, query complexity, error costs, and changing information needs.
Section by section
Introduction
The introduction defines RAG, shows why access to a correct source does not guarantee a grounded answer, and frames reliability as an engineering problem.
- RAG and other answer systems: Distinguish RAG from search engines and standalone language models: it retrieves external information and uses that evidence to generate an answer.
Why it matters: This distinction explains what RAG is meant to add to an AI answer. - Source access versus grounding: Explain what the Air Canada chatbot case illustrates about the difference between having the correct document nearby and giving an answer grounded in it, without claiming the chatbot used retrieval.
Why it matters: A linked or available policy cannot make a contradictory answer reliable. - Failure-focused RAG engineering: Explain why reliable RAG requires attention to failure modes rather than assuming a simple proof of concept will work consistently. Describe the introduction's CRAG example as checking retrieved documents before generation and responding when they are weak.
Why it matters: Checking the evidence addresses a concrete way an apparently simple RAG system can produce unreliable answers.
Key terms: Retrieval-Augmented Generation (RAG), external knowledge sources, grounded, Air Canada, bereavement fares, failure modes, Corrective RAG (CRAG), retrieve-then-audit
1.1 How this book teaches RAG
The section explains why the book approaches RAG through enduring research principles, failure prevention, measurement before added complexity, and cost control.
- Failure-focused research literacy: Explain how recognizing persistent architectural patterns and production failure modes helps teams choose research-backed fixes instead of guessing.
Why it matters: A paper is most useful when you can connect its idea to a failure your system actually has. - Measurement before added complexity: Justify starting with the simplest pipeline that meets an accuracy target and adding components only in response to measured failures.
Why it matters: A measured baseline lets you tell whether a more complicated pipeline is worth keeping. - Three evaluation dimensions: Distinguish faithfulness, answer relevance, and context relevance as ways to assess a RAG pipeline without reference answers.
Why it matters: Separate measures help reveal what improved—and what did not—when a pipeline changes. - RAG cost guardrails: Describe why advanced RAG systems need resource controls and name examples given in the section.
Why it matters: An improvement is less attractive if its resource costs grow unchecked.
Key terms: research literacy, persistent architectural patterns, failure modes, failure patterns, progressive complexity, accuracy bar, measured failure, faithfulness, answer relevance, context relevance
1.2 How do RAG systems work?
The section explains how RAG prepares a searchable knowledge base, retrieves and ranks evidence for a question, and uses that evidence to generate a grounded answer.
- Retrieval and generation together: Explain why RAG combines retrieval with language-model generation rather than relying on either alone. Describe how external sources can make a synthesized answer current, grounded, and attributable.
Why it matters: This explains the benefit RAG is meant to provide over search results or an unsupported model answer. - Offline knowledge base preparation: Describe what the indexing pipeline does before a user asks a question and distinguish it from the real-time query pipeline.
Why it matters: Prepared, searchable documents give the real-time pipeline a knowledge base to search. - Real-time query sequence: Trace the four stages of the query pipeline and state the purpose of each stage.
Why it matters: The sequence shows how a question becomes a response supported by selected sources. - Pipeline stages and failure points: Explain how the section relates different RAG failure patterns to the stages that can address them.
Why it matters: It helps distinguish a problem with finding evidence from a problem with using evidence.
Key terms: RAG, parametric knowledge, source knowledge, source attribution, indexing pipeline, chunking, searchable representations, specialized database, query processing, retrieval
1.3 How RAG research addresses failure points
The section uses the Air Canada failure to identify three production risks that motivate RAG research.
- Three production AI pitfalls: Explain the knowledge boundary, hallucination challenge, and private knowledge gap, and relate them to the risks of deploying a generic chatbot.
Why it matters: These pitfalls explain why a plausible chatbot answer can still be unsafe or costly in production.
Key terms: knowledge boundary, hallucination challenge, private knowledge gap, grounded
1.3.1 The knowledge boundary problem
This section explains why a trained LLM cannot know newer information and why retrieving current knowledge improves its decisions.
- Training cutoff and knowledge boundary: Explain that an LLM’s knowledge boundary is where its training-based world knowledge ends, so it cannot know the outcome of an event that happened after training.
Why it matters: A fluent answer may still be missing information that did not exist when the model was trained. - External retrieval for current knowledge: Explain why an AI assistant must actively retrieve changing organizational knowledge rather than rely only on its trained parameters, and how access to current information can improve decisions.
Why it matters: Retrieval lets an assistant use developments such as a recent port closure or court ruling when they matter to a decision.
Key terms: knowledge boundary, training cutoff, external retrieval, parametric knowledge, frozen weights or parameters
1.3.2 The hallucination challenge
This section explains why plausible but ungrounded LLM answers are dangerous, distinguishes types of hallucination, and introduces evidence checking as an incomplete safeguard.
- Plausible but ungrounded answers: Explain what a hallucination is, why it can be difficult to notice, and why a larger model does not necessarily solve that problem.
Why it matters: A convincing answer is not, by itself, evidence that its claims are true. - Types of hallucination: Distinguish the hallucination types named in this section by what goes wrong in each case.
Why it matters: Naming the kind of error helps pinpoint what an answer needs to be checked against. - Consequences of unverified synthesis: Describe why combining information without reliable grounding is risky in high-stakes applications and in multi-step systems.
Why it matters: Small unsupported claims can have serious consequences when people act on them or other system steps repeat them. - Evidence checking and its limits: Distinguish spotting a suspicious answer without evidence from checking it against retrieved evidence, and state the limit of those checks.
Why it matters: Evidence checks offer a stronger basis for scrutiny than answer appearance alone, without guaranteeing correctness.
Key terms: hallucination, unverifiable synthesis, factual hallucination, citation hallucination, logical hallucination, intrinsic, extrinsic, temporal hallucination, false consensus, Red Flagging
1.3.3 The private knowledge access problem
This section explains why enterprise AI needs access to changing private knowledge while respecting limits on policy reasoning and data access.
- Private knowledge access gap: Explain why a model trained on public internet data cannot, by itself, answer questions that depend on an organization’s private or specialized information.
Why it matters: An answer can miss the information an organization actually uses to make decisions. - Changing knowledge and reuse: Distinguish fine-tuning on historical logs from using RAG when policies change, and describe how the same RAG workflows can serve different knowledge bases.
Why it matters: It explains why continually changing organizational knowledge should not simply be baked into model weights. - Policy application limits: Describe the model limitation this section identifies when applying policies involving dates or precise calculations.
Why it matters: Retrieving the right policy does not guarantee that a model can apply every rule correctly. - Granular private data controls: Explain why access to private organizational data requires controls outside model weights, including restrictions on who can access which information.
Why it matters: An enterprise system must control access to confidential information, not merely make that information available to a model.
Key terms: private knowledge, proprietary, specialized and curated information, fine-tuning, historical logs, RAG systems, knowledge bases, spatiotemporal reasoning, retroactive dates, precise discount calculations
1.3.4 RAG is an enduring architectural pattern
This section explains why RAG remains useful as longer context windows, fine-tuning, and caching improve: it selects relevant external information for a model rather than replacing those techniques.
- RAG's enduring role: Explain why RAG remains an architectural pattern despite advances in other techniques. Describe its core function as connecting generative models to external data and providing the right context for a task.
Why it matters: This distinction helps you evaluate new techniques as possible additions to RAG rather than automatic replacements for it. - Retrieval versus longer context: Distinguish the benefit of longer context windows from the benefit of retrieving relevant information first. Explain why putting a large collection into a prompt can be less effective than selecting the pertinent parts.
Why it matters: A bigger prompt is not a substitute for finding the information that matters. - RAG as context engineering: Describe how RAG constructs useful context for a model: it selects relevant chunks, puts them in the prompt, and filters out irrelevant information rather than changing the model's weights.
Why it matters: Thinking of RAG as context engineering focuses attention on what information reaches the model. - Fine-tuning alongside retrieval: Explain how fine-tuning can improve RAG components or their use of retrieved context without replacing access to external knowledge. Describe the remaining difficulty of combining general ability with specialized organizational knowledge.
Why it matters: You can use training to improve a RAG system while keeping its way of accessing and updating external knowledge. - Caching within RAG: Explain why caching is presented as a performance optimization within RAG rather than an alternative to retrieval-augmented generation.
Why it matters: A faster or cheaper implementation can still rely on RAG's core retrieval-and-generation pattern.
Key terms: architectural pattern, external data, knowledge boundaries, hallucination risks, private knowledge, longer context windows, retrieval-first approach, precision, lost in the middle, context engineering
1.3.5 Why RAG justifies infrastructure investment
This section gives practical reasons to invest in RAG infrastructure, identifies cases where retrieval may not be worth it, and cautions against taking reported results at face value.
- Changing and rare knowledge: Explain why retrieval is useful when facts change faster than a model can be retrained or are too rare to have been learned reliably.
Why it matters: This helps you recognize when storing knowledge outside the model is worth the infrastructure. - Grounding and source support: Distinguish using a passage to improve an answer from returning it so a person can check the answer, and explain why a citation alone does not establish support.
Why it matters: Checkable evidence can save review time, but reviewers still need to confirm that it supports the claim. - Private knowledge in owned indexes: Describe why an owned index can be useful when the knowledge cannot go into a public model's training set.
Why it matters: Where information may be stored can determine whether a RAG approach is viable. - Retrieval costs and synthesis value: Justify when retrieval can be economical and why RAG can deliver value beyond keyword search despite added infrastructure costs.
Why it matters: The investment case depends on both the cost of producing answers and the work the system saves people. - Cases against adding retrieval: Describe situations in which skipping retrieval may be better and the risk of adding a RAG pipeline.
Why it matters: Adding a retriever is a design choice to test, not an automatic improvement. - Caution with reported results: Explain why reported RAG results should be read cautiously, including results in this book.
Why it matters: This caution helps you avoid treating a reported improvement as proof that RAG will pay off in your setting.
Key terms: knowledge boundary, weights, retrieval, fine-tuning, grounding, context engineering, provenance, attribution, checkability, private knowledge
1.4 Failure points within RAG systems
This section introduces a taxonomy of RAG failures and shows how it can guide diagnosis and testing.
- Three layers of RAG failure: Distinguish retrieval, augmentation, and generation failures using the examples given in the section.
Why it matters: Separating these layers helps you locate a problem instead of treating every bad answer as the same kind of failure. - Failure taxonomy for diagnosis: Explain how a catalog of specific failure points makes production RAG problems easier to diagnose and prioritize.
Why it matters: A named failure point gives a team a more actionable starting place than a general report of bad answers. - Failure targeted evaluation: Explain why measurement frameworks and test suites should target individual RAG failure points before production.
Why it matters: Testing against known failure points can reveal problems before users encounter them.
Key terms: retrieval, augmentation, generation, three-layer taxonomy, seven failure modes, failure points, systematic diagnosis, FP2: Missed the Top Rank, systematic evaluation, measurement frameworks
1.4.1 RAG failure examples
This section uses public and hypothetical RAG failures to show why plausible answers and functioning components are not enough: reliability also requires detecting ungrounded answers and remaining reliable after index changes.
- Retrieved Source Quality: Explain how a RAG system can retrieve material successfully yet produce a harmful answer when its sources are jokes or satire.
Why it matters: Finding a source is not the same as finding a source worth repeating. - Ranking And Answer Specificity: Trace the loan-officer example from the buried current memo to the generic answer, distinguishing the retrieval-ranking failure from the answer-specificity failure.
Why it matters: Having the right document somewhere in an index does not ensure the user gets the right advice. - Fabrication Despite Relevant Documents: Explain why retrieved documents do not eliminate hallucinations, using the fabricated compliance advice as an example.
Why it matters: A detailed, authoritative-sounding answer still needs to be checked against its evidence. - Silent Answer Failure Points: Distinguish missing content, content cut from context, overlooked text, and incomplete answers as ways a RAG response can fail without an obvious error.
Why it matters: A fluent response can conceal several different failures that require different diagnoses. - Confidence Calibration And Trust: Explain how confidently delivered high-stakes mistakes create a trust barrier even when most answers are correct.
Why it matters: Reliability affects whether people can use the assistant efficiently, not just its answer score. - Measurable RAG Reliability: Describe the section's definition of reliability, distinguishing an observed failure from its underlying fault and explaining why provenance alone cannot detect silent degradation.
Why it matters: A system can keep passing component health checks while quietly delivering wrong answers.
Key terms: corpus, retrieval, Missed the Top Rank, Incorrect Specificity, 6-Month Fast-Track, Factually Inconsistent Hallucination, grounding, source material, Missing Content, Not in Context
1.4.2 Research-backed solutions
The section links recurring RAG failures to research-backed architectural responses rather than treating them as reasons to abandon RAG.
- Failure-driven architectural research: Explain why the failures described here motivate changes to RAG architecture rather than prompt wording or abandonment of RAG.
Why it matters: It frames a failed answer as a clue about which part of the system needs attention. - Checks for contradictory answers: Distinguish the section's two research responses to an answer that contradicts available evidence: CRAG evaluates retrieved documents before generation and can fall back to web search, while Self-RAG checks whether generated claims are supported.
Why it matters: The two approaches target evidence problems at different points in producing an answer. - Other failure response pairings: Describe which failures motivate HyDE, RAPTOR, and compression work, using only the brief responses given here.
Why it matters: Matching a technique to the observed failure is more useful than treating every bad answer as the same problem.
Key terms: failure points, architectural challenges, CRAG, Self-RAG, evaluator, supported, HyDE, RAPTOR, compression, failure point 2
1.5 Naive, advanced, modular, and agentic RAG
This section introduces RAG architectures as stages of increasing complexity and explains the boundary between modular and agentic RAG.
- Modular and agentic boundary: Distinguish modular from agentic RAG by who determines the possible steps, and explain the resulting difference in cost predictability.
Why it matters: This distinction helps you recognize when a branching pipeline is still modular rather than agentic. - Architectural stages as planning aid: Explain why the section presents RAG architectures as four stages of increasing complexity.
Why it matters: Thinking in stages helps match architectural complexity to a system's changing requirements.
Key terms: modular RAG, agentic RAG, routers, retries, critique steps, architectural patterns, four stages, upgrade triggers
1.5.1 Naive RAG: Establishing the foundation
Introduces naive RAG’s retrieve-then-generate pipeline, where it is useful, and why its simple retrieval can fail.
- Naive RAG pipeline: Trace how naive RAG turns documents and a query into a generated answer.
Why it matters: This simple pipeline is the baseline against which more elaborate RAG approaches can be understood. - Benefits and suitable uses: Explain why the simple pipeline is attractive and identify situations in which this section says it can suffice.
Why it matters: A basic pipeline can be a sensible choice when its simplicity fits the task. - Retrieval limits and mitigations: Explain how naive RAG can miss needed context and state the limitations and mitigations identified here.
Why it matters: If retrieval misses the right passage, the generator may not have the evidence it needs, even when the answer exists in the collection.
Key terms: Naive RAG, retrieve-then-generate, chunks, embeddings, retrieved context, Rapid prototyping, Less complexity, Low cost, Clear evaluation, top k
1.5.2 Advanced RAG: Optimizing the components
This section explains how advanced RAG improves a basic pipeline through targeted component optimizations without changing its fundamental architecture.
- Advanced RAG architecture: Explain how advanced RAG relates to the basic RAG pipeline and what kinds of improvement it pursues.
Why it matters: This distinguishes component optimization from replacing the basic RAG architecture. - Document and query preparation: Describe how preprocessing and query enhancements improve what is available to retrieval and how questions are sent to it.
Why it matters: Improving documents and questions can help retrieval before its results are even ranked. - Retrieval quality refinements: Describe the retrieval improvements listed here and how the section’s multi-query example combines them.
Why it matters: Retrieval quality depends not just on finding candidates but also on combining and refining them. - Retrieved context optimization: Explain why advanced RAG compresses, deduplicates, and orders retrieved context for the generator.
Why it matters: Finding useful passages is not enough if the generator cannot effectively use the context it receives. - Advanced RAG investment costs: Explain what additional evaluation and operational investment these optimizations require.
Why it matters: The quality gains have costs that a team must plan to evaluate and sustain.
Key terms: Advanced RAG, basic pipeline, preprocessing optimization, semantic coherence, metadata, multimodal content, query expansion, HyDE, hybrid search, dense retrieval
1.5.3 Modular RAG: Adaptive and composable systems
This section explains how modular RAG combines specialized components for different needs, and why that flexibility requires investment in interfaces and routing.
- Interchangeable specialized components: Explain how modular RAG differs from a fixed pipeline by combining interchangeable components specialized for retrieval methods, content types, or tasks.
Why it matters: This is the architectural distinction that makes modular RAG adaptable to varied information needs. - Query-specific module routing: Describe how a modular system can arrange and select combinations of modules at query time according to the query or performance requirements.
Why it matters: Routing is what turns a collection of components into an adaptive system. - Independent scaling and extension: Explain why modular RAG separates components for development, deployment, and scaling, and how standard interfaces support substitution and gradual extension.
Why it matters: Independence helps teams match resources to workloads and change one component without disrupting the entire pipeline. - Orchestration investment cost: Explain where the main engineering investment lies in modular RAG, regardless of whether it uses microservices or a modular monolith.
Why it matters: The flexibility of interchangeable modules depends on reliable coordination, not just on having separate components.
Key terms: modular RAG, component modularity, composable frameworks, flexible configuration, route queries, query time, modular architecture, standardized interfaces, extensible frameworks, plugin architectures
1.5.4 Agentic RAG: Autonomous information-seeking systems
This section explains why agentic RAG controls retrieval during generation, what that autonomy enables, and why production systems often constrain it.
- Autonomous retrieval and timing: Explain how agentic RAG differs from a pipeline that retrieves once before generation, and why a reasoning model may need evidence later.
Why it matters: It explains why giving a model evidence only at the start may not support a long reasoning task. - Adaptive information gathering capabilities: Describe how agentic RAG can formulate queries, retrieve repeatedly, assess its output, and coordinate information gathering for complex tasks. Distinguish its autonomous control from critique or validation steps, which other RAG pipelines may also have.
Why it matters: These capabilities let retrieval follow the changing needs of a complex task rather than a fixed initial query. - Bounded autonomy tradeoffs: Justify when agentic RAG is useful and why teams often limit its autonomy. Describe the costs and predictability concerns and the kinds of controls the section gives.
Why it matters: Choosing agentic RAG means weighing the value of adaptive searching against its operating cost and unpredictability.
Key terms: Agentic RAG, runtime, retrieval, timing mismatch, dynamic retrieval triggering, autonomous query formulation, self-monitoring and correction, multi-step reasoning coordination, bounded autonomy, token costs
1.5.5 RAG evolution
This section frames the choice among four RAG approaches as a business decision and cautions that its suggested thresholds are rules of thumb.
- Business-driven RAG architecture choice: Distinguish when the section recommends naive, advanced, modular, or agentic RAG based on constraints such as retrieval quality, query complexity, error costs, and changing information needs.
Why it matters: The right architecture depends on the problem and its costs, not just on how capable the architecture sounds. - Status of selection thresholds: Explain how the section says to use its numerical thresholds when making an architecture choice.
Why it matters: Treating illustrative thresholds as universal cutoffs could lead to the wrong investment decision.
Key terms: naive RAG, advanced RAG, modular RAG, agentic RAG, business constraints, thresholds, rules of thumb
1.6 Summary
Where this is used later
- The retrieval evaluator (chapter 11) · The retrieval evaluator develops the document-checking idea introduced by the CRAG example.
- Mapping failure modes to RAG approaches (chapter 15) · Supports selecting RAG approaches according to observed failure modes.
- The RAGAS evaluation framework (chapter 14) · Develops the three evaluation dimensions introduced here.
- The efficiency-quality tradeoff (chapter 15) · Revisits the tradeoff between efficiency and quality.
- Dynamic knowledge retrieval (chapter 6) · Dynamic knowledge retrieval addresses the need for information beyond training-based knowledge.
- (chapter intro) (chapter 2) · The problem of unsupported or hallucinated answers in RAG.
- Mapping scores to failure points (chapter 14) · Helps relate observed answer errors to failure points.
- Anatomy of a single-pass failure (chapter 11) · Provides context for examining how an assistant's answer can fail.
Concepts explained in this chapter
- retrieval-augmented generationRetrieval-augmented generation (RAG) supplies retrieved external information to a language model before it generates a response.
- agentic RAGAgentic RAG is a retrieval-augmented generation architecture in which runtime decisions control retrieval, reasoning, or tool actions based on system state and signals.
- knowledge boundaryA self-knowledge boundary is the limit distinguishing information a language model can reliably know internally from information requiring external retrieval.
- hallucinationA generated claim that contradicts the evidence or source context available to the generation system.
- architectural styleAn architectural pattern is a recurring high-level organization of software components and their interactions.
- information retrievalRetrieval finds candidate documents or passages from a knowledge base in response to a query.
- cache-augmented generationA generation approach that augments retrieval with cached information or computation to improve efficiency and reduce inference cost.
- context engineeringThe deliberate selection, organization, and presentation of information in a model's input context.
- similarity searchSimilarity search finds items whose representations are close to a query representation according to a similarity or distance function.
- failure modeA failure mode is a specific way a system or component can behave incorrectly or fail to meet its requirements.
- modular RAGModular RAG is a retrieval-augmented generation architecture composed of interchangeable specialized components that can be recombined or reconfigured.
- reflection tokenIsUse is a Self-RAG reflection token that signals the overall usefulness of a generated answer.
- source knowledgeExternal knowledge is information outside a model's learned parameters that can be supplied from an external source.
- attributionThe association of generated claims or responses with the information sources supporting them.
- retrieval rankingRanking is the ordering or filtering of retrieved candidates according to a relevance score, including ordering passages by relevance to a query.
- advanced ragAdvanced RAG is a retrieval-augmented generation architecture that optimizes retrieval, document processing, and post-retrieval components while retaining the basic pipeline.
- CRAGThe Comprehensive RAG Benchmark is a benchmark for evaluating retrieval-augmented generation systems on question-answering performance.
- evaluationSystematic evaluation is a structured process for measuring system behavior against defined criteria and test conditions.
- automated synthesisThe automated integration and summarization of information from retrieved sources into a coherent result.
- checkabilityThe property of an answer being verifiable by inspecting an identified supporting passage or source.
- groundingGrounding is conditioning a generated answer on retrieved evidence so that its claims can be checked against supporting passages.
- autonomous query formulationAutonomous query formulation is the generation of search queries by a system from its current generation context, predicted information needs, or detected knowledge gaps.
- bounded autonomyBounded autonomy is an agentic system design in which autonomous behavior is restricted by explicit operational, budgetary, or depth limits.
- chain-of-thoughtMulti-step reasoning coordination is the organization of subproblem solving, information gathering, and result synthesis across iterative reasoning cycles.
- dynamic retrieval triggeringDynamic retrieval triggering is the runtime decision to obtain additional information when generation confidence, content, or task requirements indicate a need.
- self-correctionSelf-monitoring and correction is a system capability that evaluates outputs and information quality and initiates corrective actions when problems are detected.
- parametric knowledgeParametric memory is knowledge encoded in the learned parameters of a trained model rather than retrieved from an external source.
- context windowContext length is the maximum amount of input text that a model can process in one input sequence.
- in-context learningIn-context learning is a model adaptation method in which a model learns from information provided in its prompt without modifying its internal weights.
- language modelA language model generates or evaluates sequences of natural-language tokens.
- context relevanceContext relevance is a metric measuring how much retrieved context is useful for answering the question.
- fidelityFaithfulness measures whether a generated response is supported by the retrieved context rather than introducing unsupported claims.
- relevanceAnswer relevance is a metric measuring how well a generated answer addresses the user's question.
- vocabulary mismatchVocabulary mismatch is a difference between the terms used in a query and the terms used in relevant source content.
- indexing pipelineAn indexing pipeline is an offline workflow that transforms source documents into stored representations suitable for retrieval.
- knowledge baseThe internal knowledge base is the case study's repository of Mastercard support information supplied to the assistant.
- response generationGeneration produces a natural-language response from a language model conditioned on retrieved evidence.
- modularityA modular architecture organizes a system into independently developable, deployable, scalable, and replaceable components.
- orchestration layerAn orchestration layer coordinates modules, interfaces, routing decisions, and execution across a larger software system.
- test suiteA test suite is a collection of tests designed to assess specified behaviors or failure conditions in a system.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
