Chapter 3: External knowledge and retrieval
This introduction explains why LLM-based systems use external knowledge and distinguishes retrieval, caching, and direct injection as ways to supply it.
After this chapter you will be able to
- Purpose of external knowledge. Explain why an LLM-based system supplies external knowledge in addition to the model’s training data.
- Three external knowledge strategies. Distinguish retrieval, caching, and direct injection by how each supplies external knowledge to the model.
- External Knowledge and Parametric Memory. Distinguish an LLM’s static parametric memory from external knowledge, and describe the kinds of sources external knowledge can provide.
- Grounding and Its Limits. Explain what makes a response grounded and what grounding does not guarantee.
- External Knowledge in Working Context. Describe how external knowledge is made available to an LLM, using RAG as the example and distinguishing the agent’s stated role in agentic RAG.
- Query-time external knowledge. Explain how RAG lets an LLM use retrieved external knowledge alongside what it learned during pre-training to answer questions without retraining.
- Grounding and answer reliability. Distinguish an answer supported by retrieved evidence from an accurate answer, and describe the factors this section says affect reliability.
- Two RAG workflow timelines. Distinguish the offline indexing pipeline from the online generation pipeline by when each runs and what each does.
- Offline indexing and data loading. Trace how the indexing pipeline begins: it runs independently of user queries, uses knowledge hooks to locate sources, and loads and preprocesses text before later steps.
- Document splitting into chunks. Explain why the pipeline divides large documents into smaller, coherent chunks and what aspect of retrieval the choice of chunking strategy affects.
- Embedding conversion and vector storage. Describe how chunks become embeddings and why the pipeline stores them in a vector database.
- Dense vector retrieval. Trace how a dense vector retriever finds candidate text chunks for a query.
- Retrieval selection and reranking. Explain what retrieval top-K controls and how an optional reranking stage changes the candidate order.
- Text-based prompt augmentation. Trace how retrieved chunks become evidence in the prompt and lead to a generated answer.
- Chunk size trade-off. Explain why chunks must be small enough for precise retrieval but large enough to preserve meaning, using the warranty rule and battery exception as an example.
- Foundational chunking methods. Distinguish fixed-size, recursive, and document-based chunking by where each places boundaries, and state why these methods are the recommended starting point.
- Advanced chunking approaches. Describe how semantic, hierarchical, LLM-based, and agentic chunking choose or use boundaries, and explain their shared trade-off against foundational methods.
- Sparse Retrieval and BM25. Explain how sparse retrieval finds chunks using query-word overlap and statistical weighting, and when it is a good choice. Describe how BM25 accounts for term frequency, term rarity, and document length.
- Dense Retrieval Tradeoffs. Explain how dense retrieval uses embeddings to find conceptually related chunks despite different wording, and identify its computational tradeoff.
- Scalable Vector Search. Explain how vector databases search dense embeddings and why they use approximate nearest neighbor indexing at large scale.
- Production RAG tradeoffs. Explain why RAG’s factual-grounding benefits must be weighed against production complexity and cost.
- Retrieved evidence quality limits. Describe how chunking and similarity search can fail to return complete, relevant evidence.
- Retrieval authorization boundaries. Explain why permission filtering must precede ranking and context assembly in enterprise RAG.
- RAG platform support. Explain why teams use RAG platforms and what parts of a retrieval system those platforms support.
- Knowledge graph retrieval value. Explain how a knowledge graph represents a domain and why graph RAG uses that representation instead of a flat collection of text chunks.
- Knowledge graph building blocks. Describe how nodes, edges, and properties represent a domain as a graph that can be queried and traversed.
- Labeled property graph model. Explain what a labeled property graph stores and why its flexibility comes with limited explicit domain semantics.
- RDF ontology semantics and trade-off. Explain how RDF and ontologies make domain meaning available to retrieval, including inferred facts, and state the resulting benefits and cost.
- Graph construction pipeline. Describe how Graph RAG loads external data, extracts entities and relationships, resolves aliases, and stores the result as nodes and labeled edges.
- Query-guided graph navigation. Trace how a query identifies relevant entities and relationship types, then guides traversal to a connected subgraph.
- Structured context and grounded generation. Explain how retrieved graph facts become LLM context and how the LLM uses that context to answer.
- Complementary retrieval channels. Explain why hybrid RAG combines retrieval channels and what kinds of matches lexical, dense vector, and graph retrieval contribute.
- Hybrid retrieval stages. Trace how a hybrid retriever narrows candidates, searches through parallel channels, and produces a ranked context.
- Reciprocal rank fusion. Distinguish weighted score fusion from reciprocal rank fusion and explain how RRF combines rankings.
- Agent controlled context construction. Trace how an agentic RAG system determines what information a prompt needs, chooses retrieval tools, checks whether the gathered evidence is sufficient, and produces a grounded response.
- Adaptability costs and safeguards. Explain why adaptable retrieval helps with complex information needs, and why its failure modes and overhead call for controls or a simpler approach for some queries.
- Structural indexing and navigation. Trace how vectorless RAG turns a structured document into an outline and uses that outline to find grounding text without embeddings or a vector database.
- Structured document tradeoffs. Explain why outline-based retrieval can be more precise for long structured documents and where that advantage breaks down.
- Preloaded knowledge workflow. Trace how CAG moves retrieval out of the request path by loading knowledge into a shared prompt prefix and appending new user queries to it.
- Three architectural caching layers. Distinguish the serving engine's KV cache, provider-facing prompt caching, and the application's management of source knowledge and the exact prefix.
- Cache hits and fresh answers. Explain why CAG still generates a fresh answer for each query and must give correct results even when the cached prefix is missed.
- Cache footprint and application policies. Explain how cache size constrains CAG and distinguish the memory policy from the invalidation policy.
- Direct document injection. Describe how context stuffing supplies external knowledge by placing the entire source text in the model’s context window rather than indexing and retrieving chunks. Explain that the text is cleaned and bounded before injection, preferably within the system instructions.
- Benefits and repeated input costs. Explain why omitting retrieval simplifies the architecture and avoids chunking errors and retrieval misses, while distinguishing context stuffing from CAG’s cached knowledge across requests.
- Long input trade-offs and fit. Explain the cost and comprehension trade-offs of resending large documents, and state the situations for which the section recommends context stuffing.
- Empirical architecture progression. Describe the progression from context stuffing or CAG for prototypes, to standard RAG when context size or traffic demands it, to more complex RAG architectures only when benchmarks expose specific failures.
- Sparse document indexing. Explain how the example uses a TF-IDF vectorizer to turn documents into sparse numerical vectors for keyword-based retrieval, and why this is simpler than the dense retrieval often used in production RAG.
- Retrieval and grounded generation. Trace what happens after a user asks a question: vectorize the query with the same vectorizer, rank documents by cosine similarity, put the highest-scoring matches into a context-only prompt, and send that prompt to the model.
- Local open-source RAG components. Explain how this example replaces the proprietary generation model with one downloaded through Hugging Face Transformers while retaining the previous example’s TF-IDF-based retrieval.
- Local RAG component roles. Describe how Ollama, Qdrant, and Agno fit together in the example: Ollama runs the local LLM and embedding model, Qdrant stores and queries document embeddings, and Agno connects the components and provides a web interface.
- RAGFlow document question workflow. Trace the example from connecting to a running RAGFlow server through preparing a document and asking a question grounded in it.
- Platform abstraction tradeoff. Explain what RAGFlow handles in this example and what is gained and obscured by using its higher-level interface instead of implementing retrieval steps manually.
- Retriever as a callable tool. Explain that the example creates a document retriever and wraps it as a tool so the agent can call the retrieval pipeline.
- Agent setup and model requirements. Describe the three inputs to LangChain's create_agent in this example and why the chosen local model must support tool calling and answering from retrieved passages.
- Query-to-answer retrieval loop. Trace how the agent handles a knowledge-base question: decide whether search is needed, call the tool with a relevant query, inspect its result, and use that context for a grounded answer.
- Logical location in PageIndex. Explain how PageIndex differs from embedding-based retrieval: it organizes documents into a hierarchical tree an AI agent can navigate to find where information belongs, rather than searching for similar text chunks.
- Shared context cache reuse. Explain how encoding shared context once lets multiple generations reuse its cached attention state instead of recomputing it.
- Cache pattern versus CAG. Distinguish this low-level KV-cache optimization from a complete CAG system, which also makes decisions about cached context and its use across interactions.
- Full document context stuffing. Explain how context stuffing differs from RAG: it skips index creation and loads the full document into the LLM’s context window.
- User and system prompt placement. Distinguish the two examples: injecting the text document into the user prompt versus injecting it into the system prompt, where it is separated from the user question and the model is instructed to act as an expert on that text.
What it builds on
- External knowledge (chapter 1) · Introduces external knowledge as a context source.
- The context window (chapter 1) · Introduces the context window into which information is placed.
- The system prompt (chapter 2) · The role of a system prompt in directing an agent.
Section by section
Introduction
This introduction explains why LLM-based systems use external knowledge and distinguishes retrieval, caching, and direct injection as ways to supply it.
- Purpose of external knowledge: Explain why an LLM-based system supplies external knowledge in addition to the model’s training data.
Why it matters: This explains why the information supplied to a model need not be limited to what it learned during training. - Three external knowledge strategies: Distinguish retrieval, caching, and direct injection by how each supplies external knowledge to the model.
Why it matters: The distinction clarifies whether outside information is found on demand, prepared for reuse, or placed directly in the context window.
Key terms: external knowledge, parametric memory, retrieval, caching, direct injection, retrieval-augmented generation (RAG), cache-augmented generation (CAG), context stuffing
3.1 External knowledge
This section defines external knowledge, explains how it supports grounded responses, and introduces retrieval as a way to place it in an LLM’s context.
- External Knowledge and Parametric Memory: Distinguish an LLM’s static parametric memory from external knowledge, and describe the kinds of sources external knowledge can provide.
Why it matters: It explains why a model may need information beyond what it learned during training. - Grounding and Its Limits: Explain what makes a response grounded and what grounding does not guarantee.
Why it matters: A traceable answer is easier to check, but its source still needs scrutiny. - External Knowledge in Working Context: Describe how external knowledge is made available to an LLM, using RAG as the example and distinguishing the agent’s stated role in agentic RAG.
Why it matters: Outside information helps answer a question only when the system brings the appropriate material into the model’s context.
Key terms: parametric memory, external knowledge, non-parametric data, grounding, grounded response, context window, working context, retrieval workflows, retrieval-augmented generation (RAG), agentic RAG
3.2 Retrieval-augmented generation
Introduces RAG as query-time use of external knowledge, explains why grounding does not guarantee accuracy, and distinguishes its offline indexing and online generation workflows.
- Query-time external knowledge: Explain how RAG lets an LLM use retrieved external knowledge alongside what it learned during pre-training to answer questions without retraining.
Why it matters: This explains what retrieval adds to an LLM without changing the model itself. - Grounding and answer reliability: Distinguish an answer supported by retrieved evidence from an accurate answer, and describe the factors this section says affect reliability.
Why it matters: A cited or retrieved passage is only as useful as the sources and process that put it in front of the model. - Two RAG workflow timelines: Distinguish the offline indexing pipeline from the online generation pipeline by when each runs and what each does.
Why it matters: The timeline distinction shows which work can happen in the background and which must fit the user's wait time.
Key terms: RAG, external knowledge, parametric memory, grounded response, grounding, freshness, authority, chunking, ranking, permission filters
3.2.1 The indexing pipeline
The section explains how an offline indexing pipeline turns external documents into searchable vectors.
- Offline indexing and data loading: Trace how the indexing pipeline begins: it runs independently of user queries, uses knowledge hooks to locate sources, and loads and preprocesses text before later steps.
Why it matters: It distinguishes preparing the knowledge base from answering an active query. - Document splitting into chunks: Explain why the pipeline divides large documents into smaller, coherent chunks and what aspect of retrieval the choice of chunking strategy affects.
Why it matters: The boundaries between chunks affect both retrieval efficiency and the context available in retrieved text. - Embedding conversion and vector storage: Describe how chunks become embeddings and why the pipeline stores them in a vector database.
Why it matters: This is how prepared chunks become searchable by meaning.
Key terms: indexing pipeline, knowledge hooks, data loading, metadata, text splitting, chunking, chunks, embeddings, embedding model, similarity-based retrieval
3.2.2 The generation pipeline
This section explains how an online RAG generation pipeline transforms a query, retrieves relevant chunks, adds their text to a prompt, and generates an answer.
- Online generation latency: Explain why the generation pipeline must operate quickly when a user or application submits a query.
Why it matters: Speed matters across the whole pipeline, not just during model inference. - Optional query transformation: Describe why the system may transform a query before retrieval and give an example of such a transformation.
Why it matters: A better-aligned query can help retrieval find relevant material even when the user's wording differs from the documents. - Dense vector retrieval: Trace how a dense vector retriever finds candidate text chunks for a query.
Why it matters: This connects the user's query to the stored knowledge that can support an answer. - Retrieval selection and reranking: Explain what retrieval top-K controls and how an optional reranking stage changes the candidate order.
Why it matters: Selecting and ordering chunks affects the evidence the model receives and the noise it must handle. - Text-based prompt augmentation: Trace how retrieved chunks become evidence in the prompt and lead to a generated answer.
Why it matters: The model receives readable retrieved evidence, not the vectors used to find it.
Key terms: generation pipeline, critical latency path, query transformation, query expansion, hypothetical document generation, subqueries, retriever, embedding, similarity search, vector database
3.2.3 Chunking strategies
This section explains how chunk size and chunking strategy affect the evidence retrieved for a RAG system, and recommends starting simply before adopting more complex methods.
- Chunk size trade-off: Explain why chunks must be small enough for precise retrieval but large enough to preserve meaning, using the warranty rule and battery exception as an example.
Why it matters: A retrieved passage can look relevant yet support the wrong answer if its chunk omits a crucial qualification. - Foundational chunking methods: Distinguish fixed-size, recursive, and document-based chunking by where each places boundaries, and state why these methods are the recommended starting point.
Why it matters: These three boundary rules give a practical starting point without the added complexity of advanced methods. - Advanced chunking approaches: Describe how semantic, hierarchical, LLM-based, and agentic chunking choose or use boundaries, and explain their shared trade-off against foundational methods.
Why it matters: The methods address different coherence and document-mix problems, but their extra cost needs a reason. - Evidence-based strategy selection: Justify starting with a foundational strategy and moving to advanced chunking only when evaluation shows inadequate retrieval quality or contextual coherence.
Why it matters: Evaluation helps avoid paying for more complex indexing without evidence that it improves the retrieved context.
Key terms: chunking, retrieval precision, contextual richness, foundational strategies, fixed-size chunking, recursive chunking, document-based chunking, semantic chunking, hierarchical chunking, LLM-based chunking
3.2.4 Retrieval models and vector databases
This section compares lexical, semantic, and hybrid retrieval, then explains how vector databases make dense search scalable.
- Sparse Retrieval and BM25: Explain how sparse retrieval finds chunks using query-word overlap and statistical weighting, and when it is a good choice. Describe how BM25 accounts for term frequency, term rarity, and document length.
Why it matters: Exact identifiers call for a different retrieval strength than loosely worded questions. - Dense Retrieval Tradeoffs: Explain how dense retrieval uses embeddings to find conceptually related chunks despite different wording, and identify its computational tradeoff.
Why it matters: Meaning-based search can find relevant material that keyword matching misses. - Hybrid Retrieval Fit: Describe why combining sparse and dense retrieval is a useful default for a corpus containing both exact reference data and descriptive content.
Why it matters: Mixed collections often need both reliable keyword matches and matches based on meaning. - Scalable Vector Search: Explain how vector databases search dense embeddings and why they use approximate nearest neighbor indexing at large scale.
Why it matters: This explains how meaning-based retrieval remains fast enough for large production collections.
Key terms: sparse retrieval, statistical lexical retrieval, BM25, term frequency, inverse document frequency, document length normalization, dense retrieval, embeddings, semantic similarity, hybrid retrieval
3.2.5 RAG in production environments
This section explains the quality, security, cost, and maintenance tradeoffs of running RAG in production and gives checks for managing them.
- Production RAG tradeoffs: Explain why RAG’s factual-grounding benefits must be weighed against production complexity and cost.
Why it matters: It helps you assess RAG as an operating system, not just an answer-quality improvement. - Retrieved evidence quality limits: Describe how chunking and similarity search can fail to return complete, relevant evidence.
Why it matters: It explains why retrieving apparently similar chunks can still leave an answer poorly grounded. - Retrieval authorization boundaries: Explain why permission filtering must precede ranking and context assembly in enterprise RAG.
Why it matters: A relevant document is not safe to retrieve unless the requester is allowed to access it. - Corpus and index lifecycle: Describe how document changes and embedding-model upgrades affect a production RAG index.
Why it matters: An index needs a maintenance plan to remain current as sources and models change. - Retrieval evaluation and visibility: Explain how to check for retrieval regressions and distinguish retrieval failure from failure to use retrieved evidence.
Why it matters: These checks help reveal regressions and pinpoint where an evidence failure occurred. - Retrieved context budget: Explain why the amount of retrieved context should be tuned rather than maximized.
Why it matters: A measured context budget can avoid extra cost and latency without assuming more text means better answers.
Key terms: factual grounding, production RAG stack, operational and compute costs, information fragmentation, similarity search limitations, top-K, near-duplicate passages, authorization boundaries, access control policies, permission filtering
3.2.6 RAG tools
This section explains how RAG platforms help teams build retrieval systems without implementing every component from scratch.
- RAG platform support: Explain why teams use RAG platforms and what parts of a retrieval system those platforms support.
Why it matters: It helps you recognize what a RAG platform can provide and what supporting components still matter.
Key terms: RAG platforms, ingestion, indexing, retrieval orchestration, evaluation, serving engines
3.3 Knowledge graphs
The section explains how knowledge graphs organize external context as connected entities and relationships, and why graph RAG uses that structure for retrieval.
- Knowledge graph retrieval value: Explain how a knowledge graph represents a domain and why graph RAG uses that representation instead of a flat collection of text chunks.
Why it matters: Graph structure helps retrieval retain relationships that isolated text fragments may miss.
Key terms: knowledge graph, entities, relationships, graph RAG, linked relationship paths
3.3.1 Graph structure and modeling
This section explains the elements of a knowledge graph and compares labeled property graphs with RDF-based graphs for representing and retrieving domain knowledge.
- Knowledge graph building blocks: Describe how nodes, edges, and properties represent a domain as a graph that can be queried and traversed.
Why it matters: These elements are the vocabulary for explaining what a knowledge graph stores. - Labeled property graph model: Explain what a labeled property graph stores and why its flexibility comes with limited explicit domain semantics.
Why it matters: This distinction helps explain what a property graph does—and does not—make explicit about a domain. - RDF ontology semantics and trade-off: Explain how RDF and ontologies make domain meaning available to retrieval, including inferred facts, and state the resulting benefits and cost.
Why it matters: RDF's explicit semantics can improve the context retrieved for a model, but they require more work to model and operate.
Key terms: nodes, edges, properties, knowledge graph, labeled property graph, linked facts, edge labels, resource description framework (RDF), triples, formal ontologies
3.3.2 Graph RAG
This section explains how Graph RAG builds a knowledge graph from external data, navigates it to retrieve connected facts, and gives those facts to an LLM for grounded answers.
- Graph construction pipeline: Describe how Graph RAG loads external data, extracts entities and relationships, resolves aliases, and stores the result as nodes and labeled edges.
Why it matters: The quality of the graph determines which connected facts the system can later retrieve. - Query-guided graph navigation: Trace how a query identifies relevant entities and relationship types, then guides traversal to a connected subgraph.
Why it matters: Following relationships lets retrieval collect facts connected to the question rather than just isolated matches. - Structured context and grounded generation: Explain how retrieved graph facts become LLM context and how the LLM uses that context to answer.
Why it matters: Retrieval helps only if the connected facts are presented clearly enough for the LLM to use. - Vector search and graph roles: Distinguish flexible semantic matching from graph traversal and describe how a vector index can provide a starting point for traversal.
Why it matters: The distinction explains why semantic search and relationship-based retrieval can complement each other.
Key terms: graph construction pipeline, information extraction (IE), entity resolution, nodes, labeled edges, entity detection, relationship types, graph traversal, connected subgraph, augmentation
3.4 Hybrid RAG
This section explains how hybrid RAG combines retrieval methods, processes and merges their results, and where its fixed workflow falls short.
- Complementary retrieval channels: Explain why hybrid RAG combines retrieval channels and what kinds of matches lexical, dense vector, and graph retrieval contribute.
Why it matters: The choice of retrieval channels determines which kinds of relevant evidence the system can find. - Hybrid retrieval stages: Trace how a hybrid retriever narrows candidates, searches through parallel channels, and produces a ranked context.
Why it matters: This sequence shows how separate search results become usable context for generation. - Reciprocal rank fusion: Distinguish weighted score fusion from reciprocal rank fusion and explain how RRF combines rankings.
Why it matters: RRF offers a way to merge unlike retrieval results without tuning their raw scores onto a shared scale. - Fixed workflow limitation: Describe the reach and limitation of the multi-channel hybrid retrieval described here.
Why it matters: Finding more kinds of evidence does not by itself make a retrieval workflow adaptable.
Key terms: Hybrid RAG, lexical search, dense vector search, knowledge graphs, hybrid search, hybrid graph RAG, metadata pre-filtering, multi-channel retrieval, fusion and ranking, BM25
3.5 Agentic RAG
This section explains how an agent plans and iterates over retrieval to build grounded context, then describes common agentic RAG patterns and their tradeoffs.
- Agent controlled context construction: Trace how an agentic RAG system determines what information a prompt needs, chooses retrieval tools, checks whether the gathered evidence is sufficient, and produces a grounded response.
Why it matters: The agent’s retrieval decisions, not just the final prompt, determine what evidence reaches the answer. - Agentic retrieval patterns: Distinguish routing, query-planning, iterative, and autonomous orchestration agents by the retrieval decision each pattern emphasizes.
Why it matters: These names identify different ways to give an agent control over retrieval. - Adaptability costs and safeguards: Explain why adaptable retrieval helps with complex information needs, and why its failure modes and overhead call for controls or a simpler approach for some queries.
Why it matters: Agentic retrieval is useful when adaptation earns its cost, but it needs bounds to remain reliable and efficient.
Key terms: agentic RAG, context construction, retrieval tools, grounded final response, routing agents, query-planning agents, iterative agents, autonomous orchestration agents, adaptability, heterogeneous knowledge sources
3.6 Vectorless RAG
This section explains how vectorless RAG uses document structure for retrieval, where it works well, and how it can complement vector RAG.
- Structural indexing and navigation: Trace how vectorless RAG turns a structured document into an outline and uses that outline to find grounding text without embeddings or a vector database.
Why it matters: The document outline provides a route to relevant source text without similarity search. - Structured document tradeoffs: Explain why outline-based retrieval can be more precise for long structured documents and where that advantage breaks down.
Why it matters: These tradeoffs determine whether a document outline is a useful retrieval guide. - Two-tier retrieval strategy: Describe how a production system can combine vector RAG and vectorless RAG when searching a large repository.
Why it matters: The two methods can handle different levels of the same search rather than competing as all-or-nothing choices. - Repository-scale structural knowledge: Describe how the section extends structural navigation from one document to a repository authored for agents.
Why it matters: Authored structure can make a repository navigable in the same broad way as a structured document.
Key terms: vectorless RAG, reasoning-based retrieval, reasoning tree, structural parsing, grounding context, section context, vector similarity search, fixed-size chunking, document structural quality, two-tier hybrid strategy
3.7 Cache-augmented generation
This section explains how cache-augmented generation reuses a preloaded knowledge prefix, how its caching layers work, and when its benefits justify its costs.
- Preloaded knowledge workflow: Trace how CAG moves retrieval out of the request path by loading knowledge into a shared prompt prefix and appending new user queries to it.
Why it matters: Moving retrieval off the request path changes what the system must prepare before users ask questions. - Three architectural caching layers: Distinguish the serving engine's KV cache, provider-facing prompt caching, and the application's management of source knowledge and the exact prefix.
Why it matters: Separating the layers clarifies which savings come from the serving engine and which responsibilities remain with the application. - Cache hits and fresh answers: Explain why CAG still generates a fresh answer for each query and must give correct results even when the cached prefix is missed.
Why it matters: This distinction prevents treating a performance optimization as a store of ready-made answers. - Suitable knowledge and workload: Describe when CAG works well and why an overly broad cached prefix can hurt response quality.
Why it matters: A reusable prefix pays off only if it is both repeatedly useful and relevant. - Cache footprint and application policies: Explain how cache size constrains CAG and distinguish the memory policy from the invalidation policy.
Why it matters: Without explicit size and freshness policies, a reusable prefix can become expensive or outdated.
Key terms: cache-augmented generation, CAG, cached prefix, load pipeline, KV cache, prompt caching, context caching, application-level state management, cache hit, cache miss
3.8 Context stuffing
This section explains how context stuffing puts whole source documents into a prompt, its trade-offs, and when it is a practical alternative to retrieval.
- Direct document injection: Describe how context stuffing supplies external knowledge by placing the entire source text in the model’s context window rather than indexing and retrieving chunks. Explain that the text is cleaned and bounded before injection, preferably within the system instructions.
Why it matters: This is the defining difference between context stuffing and a retrieval workflow. - Benefits and repeated input costs: Explain why omitting retrieval simplifies the architecture and avoids chunking errors and retrieval misses, while distinguishing context stuffing from CAG’s cached knowledge across requests.
Why it matters: The simpler architecture comes with repeated input rather than reuse of cached knowledge. - Long input trade-offs and fit: Explain the cost and comprehension trade-offs of resending large documents, and state the situations for which the section recommends context stuffing.
Why it matters: These trade-offs help determine when direct injection is preferable to building retrieval infrastructure.
Key terms: context stuffing, context-augmented generation, context window, system instructions, retrieval step, chunking errors, retrieval misses, CAG, cached state, per-request token costs
3.9 Choosing an approach
This section gives an evidence-based way to choose an external knowledge architecture by balancing build effort, query costs, retrieval precision, and observed failures.
- Architecture selection trade-offs: Explain that external knowledge approaches differ in offline build complexity, runtime query cost, and retrieval precision, as well as their typical use cases and failure modes.
Why it matters: These trade-offs help you compare approaches against the needs of a particular application. - Empirical architecture progression: Describe the progression from context stuffing or CAG for prototypes, to standard RAG when context size or traffic demands it, to more complex RAG architectures only when benchmarks expose specific failures.
Why it matters: An empirical progression avoids paying for complexity before there is evidence that it is needed.
Key terms: offline build complexity, runtime query cost, retrieval precision, failure modes, context stuffing, CAG, standard RAG, hybrid RAG, graph RAG, agentic RAG
3.10 Hands-on
This section introduces hands-on examples of integrating external knowledge into LLM workflows and points readers to the companion code.
3.10.1 A minimal RAG implementation with an OpenAI model
The section explains a minimal RAG example that indexes documents with TF-IDF, retrieves relevant matches, and supplies them as context to an OpenAI model.
- Sparse document indexing: Explain how the example uses a TF-IDF vectorizer to turn documents into sparse numerical vectors for keyword-based retrieval, and why this is simpler than the dense retrieval often used in production RAG.
Why it matters: The choice of TF-IDF makes the example's indexing and retrieval steps easier to inspect. - Retrieval and grounded generation: Trace what happens after a user asks a question: vectorize the query with the same vectorizer, rank documents by cosine similarity, put the highest-scoring matches into a context-only prompt, and send that prompt to the model.
Why it matters: Retrieval helps the model answer from selected external context rather than relying solely on its own knowledge.
Key terms: indexing pipeline, TF-IDF vectorizer, sparse numerical vectors, keyword-based retrieval, dense retrieval, generation pipeline, cosine similarity, retrieved context, prompt template, augmented prompt
3.10.2 RAG with an open-source generation model
This section shows how to make the preceding RAG example run locally with an open-source generation model while keeping its retrieval mechanism unchanged.
- Local open-source RAG components: Explain how this example replaces the proprietary generation model with one downloaded through Hugging Face Transformers while retaining the previous example’s TF-IDF-based retrieval.
Why it matters: It identifies what changes—and what stays the same—when moving the example from proprietary models to local, open-source components.
Key terms: Hugging Face Transformers, Model Hub, scikit-learn, TF-IDF, keyword-based retrieval
3.10.3 Fully local RAG system
This section describes the components of a locally hosted RAG example and clarifies why it is not fully offline as written.
- Local RAG component roles: Describe how Ollama, Qdrant, and Agno fit together in the example: Ollama runs the local LLM and embedding model, Qdrant stores and queries document embeddings, and Agno connects the components and provides a web interface.
Why it matters: The component roles explain what makes this RAG setup locally hosted. - Offline capability qualification: Explain why the locally hosted example supports privacy or offline-oriented scenarios but is not fully offline as written: it fetches a PDF manual from a remote URL during startup before ingesting that knowledge into local Qdrant.
Why it matters: The remote PDF fetch is an important qualification when evaluating whether the example can run without network access.
Key terms: Ollama, Qdrant, Agno, embedding model, vector database, offline capabilities, PDF manual, remote URL, local Qdrant instance
3.10.4 RAG with RAGFlow
This section shows how to run a basic document-grounded RAG workflow through RAGFlow and explains the tradeoff of delegating retrieval work to a platform.
- RAGFlow document question workflow: Trace the example from connecting to a running RAGFlow server through preparing a document and asking a question grounded in it.
Why it matters: The sequence shows how the example makes an uploaded document available to a grounded chat assistant. - Platform abstraction tradeoff: Explain what RAGFlow handles in this example and what is gained and obscured by using its higher-level interface instead of implementing retrieval steps manually.
Why it matters: A compact platform workflow is convenient, but retrieval quality can still depend on decisions the interface makes less visible.
Key terms: RAGFlow server, dataset, embedding model, parse, chat assistant, higher-level API, ingestion, parsing, indexing, retrieval quality
3.10.5 Agentic RAG with LangChain
This section shows how a minimal LangChain agentic RAG system makes document retrieval available as a tool and uses it when a query needs knowledge-base context.
- Retriever as a callable tool: Explain that the example creates a document retriever and wraps it as a tool so the agent can call the retrieval pipeline.
Why it matters: The tool wrapper is what lets the agent choose to search the documents. - Agent setup and model requirements: Describe the three inputs to LangChain's create_agent in this example and why the chosen local model must support tool calling and answering from retrieved passages.
Why it matters: These inputs give the agent both a way to retrieve documents and guidance on when to do so. - Query-to-answer retrieval loop: Trace how the agent handles a knowledge-base question: decide whether search is needed, call the tool with a relevant query, inspect its result, and use that context for a grounded answer.
Why it matters: This sequence distinguishes the example's agent-directed retrieval from simply attaching documents to every query.
Key terms: document retriever, document search tool, retrieval pipeline, create_agent, chat model, tools, system prompt, tool calling, Ollama, reasoning-action loop
3.10.6 Vectorless RAG with PageIndex
Introduces PageIndex as a vectorless RAG approach that locates information through a navigable document hierarchy rather than embedding-based similarity.
- Logical location in PageIndex: Explain how PageIndex differs from embedding-based retrieval: it organizes documents into a hierarchical tree an AI agent can navigate to find where information belongs, rather than searching for similar text chunks.
Why it matters: This distinction explains what makes the example vectorless RAG rather than another embedding-based retrieval workflow.
Key terms: PageIndex, vectorless RAG, hierarchical tree, reasoning-based retrieval
3.10.7 KV-cache optimization example
This section explains a minimal KV-cache optimization pattern that reuses encoded context across generations and distinguishes it from a complete CAG system.
- Shared context cache reuse: Explain how encoding shared context once lets multiple generations reuse its cached attention state instead of recomputing it.
Why it matters: This is the optimization the example is meant to illustrate. - Cache pattern versus CAG: Distinguish this low-level KV-cache optimization from a complete CAG system, which also makes decisions about cached context and its use across interactions.
Why it matters: A working cache optimization does not, by itself, solve the architectural decisions of a CAG system. - Example library roles: Describe the roles of PyTorch and Hugging Face Transformers in the example, including how cached attention states are exposed.
Why it matters: These roles clarify which parts of the example run the model and make its cache accessible.
Key terms: KV-cache, shared context, attention state, CAG system, cached state, PyTorch, Hugging Face Transformers, past_key_values, inference mode
3.10.8 Context stuffing
This section explains context stuffing as a way to provide an entire document to an LLM without indexing it, using either the user or system prompt.
- Full document context stuffing: Explain how context stuffing differs from RAG: it skips index creation and loads the full document into the LLM’s context window.
Why it matters: This distinction explains why context stuffing is presented as a simpler alternative to RAG. - User and system prompt placement: Distinguish the two examples: injecting the text document into the user prompt versus injecting it into the system prompt, where it is separated from the user question and the model is instructed to act as an expert on that text.
Why it matters: The placement changes how the supplied document is presented alongside the user’s question.
Key terms: context stuffing, indexing pipeline, context window, user-level context, system-level context, user prompt, system prompt
3.11 Summary
Where this is used later
- (chapter intro) (chapter 4) · The role of external knowledge as a context source.
- Domain-specific applications (chapter 12) · External knowledge as a source of evidence
- RAGOps (chapter 9) · The RAG generation pipeline
- Haystack (chapter 10) · The distinction between retrieving context and using it during generation.
- Evaluation metrics (chapter 8) · Develops the evaluation metrics mentioned here as signals for retrieval quality.
- LlamaIndex (chapter 10) · Considerations for RAG in production environments.
- Privacy (chapter 9) · Connects retrieval permissions to privacy governance.
- Evaluation (chapter 8) · Develops evaluation of system behavior.
Concepts explained in this chapter
- agentic RAGAgentic retrieval-augmented generation uses an AI agent to plan, perform, and evaluate retrieval while producing an answer.
- reasoning-based retrievalVectorless retrieval-augmented generation retrieves information without vector similarity, often by navigating document structure, and supplies it as context to a language model.
- retrieval-augmented generationA retrieval-augmented generation pipeline sequences retrieval and generation operations to supply retrieved information to a generative model.
- augmentationThe augmentation stage adds retrieved information to a model’s input to provide context for generation.
- GraphRAGA retrieval-augmented generation approach that uses a graph's entities and relationships to retrieve structured context.
- hybrid retrieval-augmented generationHybrid RAG is a retrieval-augmented generation approach that combines multiple retrieval methods over structured and unstructured data.
- generation pipelineA generation pipeline is an online workflow that retrieves relevant information at query time and uses it to produce a model response.
- hybrid searchHybrid retrieval combines keyword-based and semantic retrieval to find relevant information.
- chunkingChunking is the division of large documents into smaller, coherent text segments for downstream processing and retrieval.
- key-value cacheA key-value cache stores previously computed transformer attention keys and values for reuse during generation.
- cache-augmented generationCache-augmented generation is a generation approach that preloads relevant knowledge and reuses its processed context across multiple interactions.
- indexing pipelineAn indexing pipeline is a sequence of processing steps that transforms source documents into structures suitable for retrieval.
- prompt templateA prompt template is a predefined text structure containing slots or instructions into which task-specific inputs and context are inserted.
- prompt cachingPrompt caching reuses previously computed representations of prompt context, particularly shared prefixes, to avoid processing them again.
- groundingGrounding is the practice of supplying a model with evidence that supports or constrains its generated response.
- similarity searchNearest-neighbor search, or vector similarity search, finds stored items whose vector representations are most similar to a query representation.
- context injectionContext stuffing inserts external knowledge directly into a model's context window.
- knowledge graphA graph-structured representation of entities and their relationships.
- context engineeringContext engineering is the discipline of selecting, organizing, and updating information supplied to a model in an appropriate form and at an appropriate time.
- routerA routing agent analyzes an incoming query and directs it to a specialized retrieval tool or knowledge store.
- evaluation datasetAn evaluation set is a collection of test cases and expected or judged outcomes used to assess a system's performance.
- schemaA schema is a formal specification of the permitted structure, fields, and constraints of data or outputs.
- ReActReAct is an agent reasoning pattern that alternates between internal reasoning and external actions such as tool invocation.
- cachingThe technique of storing information or computation results for reuse to reduce repeated processing or retrieval.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
