liveBook · Manning

Chapter 3: External knowledge and retrieval

What you will learn in chapter 3 of Context Engineering

This introduction explains why LLM-based systems use external knowledge and distinguishes retrieval, caching, and direct injection as ways to supply it.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains why LLM-based systems use external knowledge and distinguishes retrieval, caching, and direct injection as ways to supply it.

Key terms: external knowledge, parametric memory, retrieval, caching, direct injection, retrieval-augmented generation (RAG), cache-augmented generation (CAG), context stuffing

3.1 External knowledge

This section defines external knowledge, explains how it supports grounded responses, and introduces retrieval as a way to place it in an LLM’s context.

Key terms: parametric memory, external knowledge, non-parametric data, grounding, grounded response, context window, working context, retrieval workflows, retrieval-augmented generation (RAG), agentic RAG

3.2 Retrieval-augmented generation

Introduces RAG as query-time use of external knowledge, explains why grounding does not guarantee accuracy, and distinguishes its offline indexing and online generation workflows.

Key terms: RAG, external knowledge, parametric memory, grounded response, grounding, freshness, authority, chunking, ranking, permission filters

3.2.1 The indexing pipeline

The section explains how an offline indexing pipeline turns external documents into searchable vectors.

Key terms: indexing pipeline, knowledge hooks, data loading, metadata, text splitting, chunking, chunks, embeddings, embedding model, similarity-based retrieval

3.2.2 The generation pipeline

This section explains how an online RAG generation pipeline transforms a query, retrieves relevant chunks, adds their text to a prompt, and generates an answer.

Key terms: generation pipeline, critical latency path, query transformation, query expansion, hypothetical document generation, subqueries, retriever, embedding, similarity search, vector database

3.2.3 Chunking strategies

This section explains how chunk size and chunking strategy affect the evidence retrieved for a RAG system, and recommends starting simply before adopting more complex methods.

Key terms: chunking, retrieval precision, contextual richness, foundational strategies, fixed-size chunking, recursive chunking, document-based chunking, semantic chunking, hierarchical chunking, LLM-based chunking

3.2.4 Retrieval models and vector databases

This section compares lexical, semantic, and hybrid retrieval, then explains how vector databases make dense search scalable.

Key terms: sparse retrieval, statistical lexical retrieval, BM25, term frequency, inverse document frequency, document length normalization, dense retrieval, embeddings, semantic similarity, hybrid retrieval

3.2.5 RAG in production environments

This section explains the quality, security, cost, and maintenance tradeoffs of running RAG in production and gives checks for managing them.

Key terms: factual grounding, production RAG stack, operational and compute costs, information fragmentation, similarity search limitations, top-K, near-duplicate passages, authorization boundaries, access control policies, permission filtering

3.2.6 RAG tools

This section explains how RAG platforms help teams build retrieval systems without implementing every component from scratch.

Key terms: RAG platforms, ingestion, indexing, retrieval orchestration, evaluation, serving engines

3.3 Knowledge graphs

The section explains how knowledge graphs organize external context as connected entities and relationships, and why graph RAG uses that structure for retrieval.

Key terms: knowledge graph, entities, relationships, graph RAG, linked relationship paths

3.3.1 Graph structure and modeling

This section explains the elements of a knowledge graph and compares labeled property graphs with RDF-based graphs for representing and retrieving domain knowledge.

Key terms: nodes, edges, properties, knowledge graph, labeled property graph, linked facts, edge labels, resource description framework (RDF), triples, formal ontologies

3.3.2 Graph RAG

This section explains how Graph RAG builds a knowledge graph from external data, navigates it to retrieve connected facts, and gives those facts to an LLM for grounded answers.

Key terms: graph construction pipeline, information extraction (IE), entity resolution, nodes, labeled edges, entity detection, relationship types, graph traversal, connected subgraph, augmentation

3.4 Hybrid RAG

This section explains how hybrid RAG combines retrieval methods, processes and merges their results, and where its fixed workflow falls short.

Key terms: Hybrid RAG, lexical search, dense vector search, knowledge graphs, hybrid search, hybrid graph RAG, metadata pre-filtering, multi-channel retrieval, fusion and ranking, BM25

3.5 Agentic RAG

This section explains how an agent plans and iterates over retrieval to build grounded context, then describes common agentic RAG patterns and their tradeoffs.

Key terms: agentic RAG, context construction, retrieval tools, grounded final response, routing agents, query-planning agents, iterative agents, autonomous orchestration agents, adaptability, heterogeneous knowledge sources

3.6 Vectorless RAG

This section explains how vectorless RAG uses document structure for retrieval, where it works well, and how it can complement vector RAG.

Key terms: vectorless RAG, reasoning-based retrieval, reasoning tree, structural parsing, grounding context, section context, vector similarity search, fixed-size chunking, document structural quality, two-tier hybrid strategy

3.7 Cache-augmented generation

This section explains how cache-augmented generation reuses a preloaded knowledge prefix, how its caching layers work, and when its benefits justify its costs.

Key terms: cache-augmented generation, CAG, cached prefix, load pipeline, KV cache, prompt caching, context caching, application-level state management, cache hit, cache miss

3.8 Context stuffing

This section explains how context stuffing puts whole source documents into a prompt, its trade-offs, and when it is a practical alternative to retrieval.

Key terms: context stuffing, context-augmented generation, context window, system instructions, retrieval step, chunking errors, retrieval misses, CAG, cached state, per-request token costs

3.9 Choosing an approach

This section gives an evidence-based way to choose an external knowledge architecture by balancing build effort, query costs, retrieval precision, and observed failures.

Key terms: offline build complexity, runtime query cost, retrieval precision, failure modes, context stuffing, CAG, standard RAG, hybrid RAG, graph RAG, agentic RAG

3.10 Hands-on

This section introduces hands-on examples of integrating external knowledge into LLM workflows and points readers to the companion code.

3.10.1 A minimal RAG implementation with an OpenAI model

The section explains a minimal RAG example that indexes documents with TF-IDF, retrieves relevant matches, and supplies them as context to an OpenAI model.

Key terms: indexing pipeline, TF-IDF vectorizer, sparse numerical vectors, keyword-based retrieval, dense retrieval, generation pipeline, cosine similarity, retrieved context, prompt template, augmented prompt

3.10.2 RAG with an open-source generation model

This section shows how to make the preceding RAG example run locally with an open-source generation model while keeping its retrieval mechanism unchanged.

Key terms: Hugging Face Transformers, Model Hub, scikit-learn, TF-IDF, keyword-based retrieval

3.10.3 Fully local RAG system

This section describes the components of a locally hosted RAG example and clarifies why it is not fully offline as written.

Key terms: Ollama, Qdrant, Agno, embedding model, vector database, offline capabilities, PDF manual, remote URL, local Qdrant instance

3.10.4 RAG with RAGFlow

This section shows how to run a basic document-grounded RAG workflow through RAGFlow and explains the tradeoff of delegating retrieval work to a platform.

Key terms: RAGFlow server, dataset, embedding model, parse, chat assistant, higher-level API, ingestion, parsing, indexing, retrieval quality

3.10.5 Agentic RAG with LangChain

This section shows how a minimal LangChain agentic RAG system makes document retrieval available as a tool and uses it when a query needs knowledge-base context.

Key terms: document retriever, document search tool, retrieval pipeline, create_agent, chat model, tools, system prompt, tool calling, Ollama, reasoning-action loop

3.10.6 Vectorless RAG with PageIndex

Introduces PageIndex as a vectorless RAG approach that locates information through a navigable document hierarchy rather than embedding-based similarity.

Key terms: PageIndex, vectorless RAG, hierarchical tree, reasoning-based retrieval

3.10.7 KV-cache optimization example

This section explains a minimal KV-cache optimization pattern that reuses encoded context across generations and distinguishes it from a complete CAG system.

Key terms: KV-cache, shared context, attention state, CAG system, cached state, PyTorch, Hugging Face Transformers, past_key_values, inference mode

3.10.8 Context stuffing

This section explains context stuffing as a way to provide an entire document to an LLM without indexing it, using either the user or system prompt.

Key terms: context stuffing, indexing pipeline, context window, user-level context, system-level context, user prompt, system prompt

3.11 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 2: Instructions for AI agents Chapter 4: Tools in AI agents →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.