Chapter 5: Building knowledge bases with RAG

Introduction

Introduces knowledge bases with RAG as the next context-engineering enhancement to the basic agent and previews the chapter’s focus on vector and structure-based search.

5.1 The problem of using internal data

This section explains why agents need to find relevant information in private data and place it in context when web search cannot answer a question.

• Relevant internal data retrieval: Explain why web search is insufficient for some real-world questions and how an agent should use internal data: find the parts relevant to the current question and put them in context.
  Why it matters: It identifies the information-access problem that searching internal data is meant to solve.

Key terms: web search, internal data sources, relevant, context

5.1.1 A simple case: Single file

For a small attached file, the agent can read the whole file into the LLM’s context to answer a question about its contents.

• Whole-file context for small files: Explain how the agent can answer a question about a small attached file by reading it and placing its entire contents in the LLM’s context.
  Why it matters: A small file may not need a more elaborate retrieval method.

Key terms: file-reading tool, context, context windows

5.1.2 Multiple files

This section explains why answering a question from a multi-file archive calls for structure-based exploration instead of reading every file.

• Structure-Based File Exploration: Explain how to search an unfamiliar multi-file archive: inspect its overall structure, use file and folder names to choose promising files, read them, and explore further if needed. Explain why putting every file into the LLM context is not a workable alternative as the archive grows.
  Why it matters: This gives the agent a practical way to find relevant information without loading an entire archive into context.

Key terms: structure based, context, filenames, folder names

5.1.3 Large or extensive data

This section explains why large files and extensive document collections require selective retrieval rather than structure-based exploration alone.

• Selective retrieval for extensive data: Explain why structure-based exploration becomes impractical when a file exceeds the available context or a collection has too many, poorly organized documents, and why an agent should retrieve only the portions relevant to a question.
  Why it matters: This explains why an agent needs search when its available data becomes too large to inspect directly.

Key terms: structure-based exploration, context, selectively retrieves, vector search, keyword search

5.2 Types of search methods

Introduces the retrieval–augmentation–generation flow and explains why different search needs call for different methods.

• RAG information flow: Trace how RAG answers a question by retrieving relevant information, adding it to the LLM input alongside the question, and generating a response that uses it as a reference. Explain why this lets the LLM use information beyond its own knowledge.
  Why it matters: This flow explains how an agent can answer using current or private information that the LLM may not already have.
• Search method fit: Explain that the appropriate search method depends on the data and the question, such as whether the task calls for exact words, similar meaning, relationships, or direct examination of file structure.
  Why it matters: Recognizing what kind of match a question needs helps you choose a useful way to retrieve information.

Key terms: RAG, retrieval, augmentation, generation, knowledge base, exact word matching, semantically similar content, traversing relationships, file structure

5.2.1 Keyword search

This section explains how keyword search ranks documents, when exact matching works well, and why it misses synonyms.

• Keyword relevance scoring: Explain how keyword search uses word frequency and importance to rank documents, including the basic roles of TF-IDF and BM25.
  Why it matters: This explains why some word matches count more than others.
• Exact-match search strengths: Describe when keyword search is especially effective and give an example from the section.
  Why it matters: Exact identifiers are a strong reason to choose keyword search.
• Synonym matching limitation: Explain why keyword search can miss a relevant document that uses a synonym.
  Why it matters: A document can be relevant even when it contains none of the query’s exact words.

Key terms: keyword search, TF-IDF, term frequency, inverse document frequency, BM25, exact words or phrases, synonyms, similar expressions, synonym dictionary

5.2.2 Vector search

This section explains how vector search retrieves text by semantic similarity, why it handles natural-language queries, and when keyword search remains preferable.

• Meaning-preserving text embeddings: Explain that an embedding model turns text into numerical vectors positioned so similar meanings are close and different meanings are farther apart.
  Why it matters: This is why vector search can find relevant text that uses different words.
• Vector retrieval and similarity: Trace how stored document vectors and a newly embedded query are compared to return the closest documents. Describe what higher cosine similarity or smaller Euclidean distance indicates.
  Why it matters: It connects the idea of embeddings to how a search actually returns results.
• Semantic and contextual matching: Explain why vector search can answer natural-language questions without exact wording and how context can change the embedding of an ambiguous word.
  Why it matters: These capabilities make semantic search useful when people phrase requests differently from stored text.
• Exact matching and hybrid search: Distinguish cases suited to keyword search from semantic vector search, and explain why combining them can be useful.
  Why it matters: Choosing a search method depends on whether the task needs meaning or an exact match.

Key terms: embedding model, embedding, vector space, query vector, cosine similarity, Euclidean distance, natural-language questions, semantic search, context-aware, keyword search

5.2.3 Graph search

This section explains how graph search answers relationship-based questions and why building a useful graph remains challenging.

• Nodes edges and traversal: Explain how graph search represents entities as nodes and relationships as edges, then answers questions by traversing those connections.
  Why it matters: The graph representation explains what graph search looks through when it answers a question.
• Multistep relationship queries: Describe the kinds of questions graph search handles especially well and why traversal suits them.
  Why it matters: It helps you recognize when the connections among items are central to the question.
• Graph construction challenges: Explain the main difficulty of constructing a graph from text, how LLMs help, and what design costs remain.
  Why it matters: Easier extraction does not remove the work of designing a useful graph.

Key terms: graph search, nodes, entities, edges, relationships, graph traversal, multistep relationship queries, code-dependency analysis, unstructured text, entity extraction

5.2.4 Structure-based search

This section explains how agents can search systematically organized files by following their structure, and why hybrid search combines exact and semantic matching.

• Structure-based file exploration: Explain how structure-based search uses folders, filenames, and code elements to locate relevant information. Describe the progression from surveying a directory tree to searching for names and reading selected files, including its use with extracted archives.
  Why it matters: This gives an agent a way to find relevant files without relying on a complex search algorithm.
• Hybrid search ranking: Explain why hybrid search combines keyword and vector search, and describe how their results contribute to a final ranking.
  Why it matters: Combining the methods helps retrieve information whether a query uses an exact name or describes the idea in different words.

Key terms: structure-based search, tree, grep, filenames, folder structures, hybrid search, keyword search, vector search, normalized, weighted average

5.3 Practicing vector search

Introduces the basic idea behind vector search before the chapter’s hands-on implementation.

• Vector distance and similarity: Explain that vector search converts text into vectors and uses distances between them to express semantic similarity.
  Why it matters: This idea connects the text being searched to the distance comparisons used in vector search.

Key terms: vector search, vectors, distances, semantic similarity

5.3.1 Embedding: Converting text to vectors

This section explains how embeddings represent text as vectors and uses cosine similarity to show why vector search can find related text without exact word matches.

• Embedding and semantic distance: Explain that an embedding converts text into a high-dimensional numerical vector and that semantically similar texts tend to be close together in vector space.
  Why it matters: This is the basis for searching by meaning rather than just by words.
• Embedding function input and output: Describe how the section’s embedding function handles a single string and returns vectors obtained from the model API.
  Why it matters: It connects the idea of embeddings to the function that supplies vectors for comparison.
• Cosine similarity and meaning: Explain how cosine similarity is interpreted in the sentence comparison and why the result illustrates an advantage over keyword search.
  Why it matters: The comparison shows what embedding similarity offers when relevant text uses different words.

Key terms: embedding, vector space, semantic relationships, text-embedding-3-small, embedding vector, NumPy array, cosine similarity, keyword search, vector search

5.3.2 Chunking: Dividing long text into search units

Explains why documents are split into searchable chunks, how chunk size affects retrieval, and how overlap preserves context in fixed-length chunking.

• Reasons for document chunking: Explain why embedding and returning a whole document can be less useful than splitting it into chunks and embedding each chunk separately.
  Why it matters: Chunking lets search return a relevant part of a document rather than relying on one diluted vector for the whole document.
• Chunk size tradeoffs: Describe how choosing chunks that are too small or too large affects context and search accuracy, and how document type influences the choice.
  Why it matters: Chunk size determines whether a search result is both focused and understandable.
• Overlap across chunk boundaries: Explain why fixed-length chunks share some text with adjacent chunks and what problem that overlap mitigates.
  Why it matters: Overlap makes passages near a cut more understandable when retrieved on their own.
• Fixed-length chunk advancement: Trace how the described fixed-length chunking function selects each chunk and advances its start position, including how it stops at the document’s end.
  Why it matters: The start-position update explains both the shared text and why overlap can produce more chunks.

Key terms: embedding, vector, chunks, chunk size, context information, search accuracy, fixed-length chunking, overlap, chunk boundaries, chunk_size

5.3.3 Implementing vector search

This section implements vector search by ranking embedded chunks against an embedded query and demonstrates that it can find relevant text without exact keyword matches.

• Vector search ranking steps: Trace how vector_search converts a query to a vector, compares it with chunk vectors, ranks the results, and returns the top matches with similarity scores.
  Why it matters: These steps explain how the agent can select the most relevant chunks from a collection.
• Semantic matches beyond keywords: Explain what the short-document test demonstrates about vector search compared with exact keyword search.
  Why it matters: Relevant information need not use the same wording as the query.

Key terms: query vector, chunk vectors, cosine_similarity, argsort()[::-1], top_k, similarity score, semantic search, embeddings, keyword search

5.3.4 Why vector databases are necessary

This section explains why searching every embedding becomes impractical at scale and how vector databases address that problem, then contrasts several database options.

• Vector search scaling limits: Explain why the direct vector-search implementation is suitable for the small exercise but becomes costly with large collections, and how vector databases help.
  Why it matters: This distinction helps you recognize when a simple implementation is enough and when search needs infrastructure built for scale.
• Vector database option tradeoffs: Describe how the named vector databases illustrate different deployment and scale needs, using concrete examples from the section.
  Why it matters: The examples show that choosing a vector database depends on practical requirements, not just the need for similarity search.

Key terms: chunks, O(n), vector databases, approximate nearest neighbor (ANN), embeddings, metadata-based filtering, Chroma, Pinecone, Qdrant, Milvus

5.3.5 Exercise: Finding relevant information from web search results

This exercise uses vector search to select relevant passages from web search results and shows how doing so reduces the context sent to an LLM.

• Web result retrieval pipeline: Trace how the exercise turns full web search results into a small set of relevant chunks: collect full page content, split and embed it, then run vector search for the query.
  Why it matters: This sequence shows how to filter large search results before passing information to an LLM.
• Semantic matches beyond keywords: Explain why searching the chunks for quantum computing can find relevant material even when that exact phrase is absent.
  Why it matters: Relevant evidence may use different wording from the reader’s query.
• Relevant context and token savings: Justify selecting only the most relevant chunks instead of feeding complete search results to the LLM, and describe how the exercise measured the difference.
  Why it matters: Filtering results keeps an LLM’s context focused while avoiding the expense of including entire pages.
• Built-in search filtering option: Describe the comparable filtering available from Tavily and why the exercise still builds the pipeline explicitly.
  Why it matters: A built-in service can save implementation work, while understanding the pipeline helps when its results need adjustment.

Key terms: include_raw_content=True, fixed_length_chunking, get_embeddings, vector_search, vector search, semantic-matching capability, token count, context, savings rate, Tavily

5.4 Structure-based search

Introduces structure-based search as a way for an agent to find answers among files extracted from an archive.

• Filesystem-Guided File Exploration: Explain why an agent explores an extracted archive’s filesystem structure, then uses filenames to choose files whose contents to inspect.
  Why it matters: This gives the agent a practical way to locate relevant information when an archive contains many possible files.

Key terms: structure-based search, filesystem, filenames

5.4.1 Preparing the GAIA dataset

This subsection prepares GAIA questions and their attached files for structure-based search, including a repeatable working folder for each problem.

• GAIA questions and attachments: Explain why GAIA questions with attachments require more than loading the question data, and distinguish what load_dataset() and snapshot_download() provide.
  Why it matters: Having both the questions and their files gives the agent the material needed to answer attachment-based problems.
• Repeatable working folder: Explain why the files are copied from a cache into a working folder before each problem.
  Why it matters: A fresh working folder prevents changes made during one problem from affecting the next.
• Attachment file paths: Describe how the subsection identifies an attachment and forms its path in the working folder.
  Why it matters: The agent needs a path to the working copy of an attachment before it can access that file.

Key terms: GAIA dataset, attachments, load_dataset(), snapshot_download(), local_dir, cache, working folder, initial state, file_name, working directory

5.4.2 Implementing filesystem tools

This section explains filesystem tools that let an agent extract attachments, inspect files, and read text or media in formats suited to each file.

• Archive extraction and file discovery: Trace how an agent starts exploring a zip-file attachment: extract it, inspect the resulting folder structure, and choose files to examine.
  Why it matters: Seeing the available files first helps the agent choose where to look for an answer.
• Format-aware file reading: Describe how read_file selects a handler by file extension and presents text and tabular files to the agent.
  Why it matters: The format determines how the agent receives useful content without reading an entire long file unnecessarily.
• Question-directed media analysis: Explain why read_media_file is separate from read_file and what guides its analysis.
  Why it matters: A question gives the agent a purpose for analyzing media rather than treating it as ordinary text.
• Combined PDF text and vision: Explain why the PDF handler supplies both extracted text and page images to the LLM.
  Why it matters: Combining text and images lets the agent address PDF content that a text-only reader would miss.
• PDF analysis cost limits: Justify the implementation’s limits on PDF pages and extracted text, and state their limitation for longer documents.
  Why it matters: The limits keep the example affordable but can leave relevant content in a long PDF unread.

Key terms: unzip_file, list_files, zipfile, read_file, start_line, end_line, pandas, Markdown tables, read_media_file, PDF handler

5.4.3 Connecting tools to the agent

This section connects filesystem tools to the agent so it can explore files, including extracted zip archives, over multiple steps.

• Filesystem tool integration: Explain how the filesystem tools are made available to the agent and why its step limit is set to 20. Describe the file-exploration sequence this enables for a zip file.
  Why it matters: Connecting the tools and allowing enough steps lets the agent work through a file-based question.

Key terms: @tool decorator, filesystem tools, web search tools, max_steps

5.4.4 Solving GAIA zip file problems

This section shows how an agent solves a GAIA problem by exploring a zip attachment, reading its files, and matching job requirements to applicants.

• Zip attachment exploration sequence: Trace how the agent locates and extracts the attachment, inspects its files, reads the job requirements and applicant data, and compares them to find an answer.
  Why it matters: The sequence shows how an agent can work through a multi-file attachment to answer a question.
• Filename and format cues: Explain how filenames and extensions help the agent decide which files contain requirements and candidate data without a complex search algorithm.
  Why it matters: These cues make a simple file-structure exploration sufficient for locating the information in this example.

Key terms: unzip_file, list_files, read_media_file, read_file, filenames, extensions, format cues

5.5 Extending agents with callbacks

This section introduces callbacks as a way to extend an agent for deployment needs without changing its core code.

• Callback-based agent extension: Explain that the callback pattern can extend the chapter 4 agent’s behavior without modifying its core code, for needs such as approval before tool execution or compression of search results.
  Why it matters: It shows why deployment requirements need not force changes to the agent’s core implementation.

Key terms: callback pattern, core code, user approval, compress search results

5.5.1 The need for agent extension

This section explains how callbacks let developers extend an agent at specific points in its execution without changing its core logic.

• Callbacks as extension points: Explain what callbacks are and why they let developers add capabilities to an agent without modifying its core logic.
  Why it matters: Callbacks provide a way to customize agent behavior without rewriting the agent itself.
• Callback timing and uses: Distinguish the intervention points before and after LLM calls, before and after tool execution, and after an agent run, with an example use for each.
  Why it matters: Choosing the right intervention point lets a callback act on information when it is available.

Key terms: callbacks, extension points, agent’s core logic, before_llm_callback, after_llm_callback, before_tool_callback, after_tool_callback, after_run_callback

5.5.2 Implementing tool callbacks

This section adds callbacks to the Agent so code can intercept tool calls before execution or replace results afterward.

• Tool callback execution flow: Trace the three stages of act(): run before-tool callbacks, execute the tool only if none supplied a result, then run after-tool callbacks that may replace the result. Explain that each callback list stops at its first non-None return value.
  Why it matters: This flow explains when a tool actually runs and which result the agent receives.
• Callback registration and async support: Describe how the Agent constructor stores separate before- and after-tool callback lists, and how act() accommodates asynchronous callbacks.
  Why it matters: These details let callbacks be optional while allowing the agent to handle asynchronous ones.

Key terms: before_tool_callback, after_tool_callback, ToolResult, Early Exit pattern, before_tool_callbacks, after_tool_callbacks, inspect.isawaitable()

5.5.3 Human-in-the-loop tool-execution approval

This section shows how an approval callback lets an agent run safe tools immediately while pausing before sensitive tool calls.

• Selective approval for risky tools: Explain why human-in-the-loop approval is used for sensitive operations and distinguish tools that require confirmation from those that can run immediately.
  Why it matters: Selective approval preserves automatic execution for safe actions while giving the user control over sensitive ones.
• Approval callback return behavior: Trace how the approval callback decides whether a tool executes and what happens when the user rejects it.
  Why it matters: The callback’s return value determines whether a sensitive action actually runs and what feedback the agent receives if it does not.

Key terms: human-in-the-loop (HITL), dangerous tools, safe tools, approval callback, act(), None, tool result, rejection message

5.5.4 Compressing search results

Shows how an after-tool callback automatically compresses long web search results and how it can coexist with a before-tool approval callback without changing the agent’s core logic.

• Automatic web search compression: Trace how the after-tool callback handles a tool result: leave other tools alone, preserve short web search results, and replace long ones with the three most relevant chunks.
  Why it matters: Automatic compression saves tokens and reduces noise without repeatedly processing search results by hand.
• Search query call matching: Explain how the helper retrieves the query belonging to the particular web search result being compressed, and why it matches the tool-call ID rather than using the latest search.
  Why it matters: Matching the right query ensures compression selects chunks relevant to that specific search.
• Combined tool callback roles: Distinguish the before- and after-tool callbacks and describe how the approval and compression callbacks are registered together without changing the agent’s core logic.
  Why it matters: Separate callbacks let the agent handle approval and result compression without embedding either concern in its core logic.
• Synchronous approval limitation: Explain why the section’s synchronous HITL approval works for local testing but is unsuitable for web services or API servers.
  Why it matters: The blocking behavior marks the limit of this approval example in a deployed service.

Key terms: after_tool_callback, search_web, chunking, vector search, ToolResult, _extract_search_query, ExecutionContext, ToolCall, tool_call_id, query

5.6 Summary

The chapter summary connects the scale of internal data to appropriate search methods, reviews two retrieval approaches, and shows how callbacks extend agents without changing core logic.