5 Building knowledge bases with RAG
This chapter covers
- Why agents need external knowledge bases
- Different search methods: keyword, vector, graph, and structure-based search
- Implementing vector search from scratch with embeddings, chunking, and similarity calculation
- Building structure-based search tools for file system exploration
- Extending agent capabilities through the Callback pattern
In Chapters 2 through 4, we built the foundation of an AI agent: connecting to LLMs, implementing tool use, and creating the agent loop. Now we enter a new phase. As figure 5.1 shows, we’ll now enhance our basic agent with context engineering strategies. This chapter tackles the first of these enhancements: building knowledge bases with RAG (Retrieval-Augmented Generation).
Figure 5.1 Book structure overview - Chapter 5 in focus.
We’ll begin with the basic components of vector search: how embeddings capture text meaning, why chunking is necessary, and how vector databases operate. We’ll implement a mini vector search system to experience the full flow. Next, we extend to structure-based search using GAIA benchmark zip file problems, where the agent navigates folder structures and reads files like a human developer. Finally, we introduce the Callback pattern for extending agent behavior, implementing Human in the Loop approval and automatic search result compression.