5 Building knowledge bases with RAG
This chapter covers
- Understanding why agents need external knowledge bases
- Using keyword, vector, graph, and structure-based search methods
- Implementing vector search from scratch with embeddings, chunking, and similarity calculations
- Building structure-based search tools for filesystem exploration
- Extending agent capabilities through the callback pattern
We’ve built the foundation of an AI agent: connecting to large language models (LLMs), implementing tool use, and creating the agent loop. Now we enter a new phase. As figure 5.1 shows, we’ll enhance our basic agent with context-engineering strategies. This chapter tackles the first of these enhancements: building knowledge bases with retrieval-augmented generation (RAG).
Figure 5.1 Book structure overview: chapter 5 in focus
We’ll begin with the basic components of vector search: how embeddings capture text meaning, why chunking is necessary, and how vector databases operate. We’ll implement a mini vector search system to experience the full flow. Next, we’ll extend this system to structure-based search using General AI Assistants (GAIA) benchmark zip file problems, in which the agent navigates folder structures and reads files like a human developer. Finally, we’ll see how to use the callback pattern to extend agent behavior, as well as to implement human-in-the-loop (HITL) approval and automatic search result compression.