chapter six

6 Working with memory and knowledge RAG for agents

 

This chapter covers

  • Understanding retrieval in AI applications
  • Vector databases and similarity search
  • Building practical knowledge of RAG agents
  • Adding memory to agents with MCP

The agents we have built so far are bounded by what is already in their training data. They cannot answer questions about a document the user uploaded yesterday, cite a policy that was published last week, or remember what they discussed with the user in their last conversation. These are real limits, and they show up the moment an agent has to work with information the model has not seen.

This chapter is about closing those gaps. We start with retrieval, the mechanism for finding relevant information from outside the model’s training and bringing it into the agent’s context at the right moment. Retrieval is what enables an agent to answer questions about documents, policies, codebases, and other external knowledge sources.

From retrieval, we move to memory, which is retrieval applied to the agent’s own past experiences. Memory is what lets an agent remember user preferences across sessions, recall what it tried on a previous attempt, and build context that compounds rather than resets. Together, retrieval and memory turn an agent from a stateless reasoner into a system that can know things and remember them.

6.1 Understanding retrieval in AI applications

6.1.1 The basics of RAG

6.1.2 Delving into semantic search and document indexing

6.1.3 Applying vector similarity search

6.2 Vector databases and similarity search

6.2.1 Demystifying document embeddings

6.2.2 Querying document embeddings from Chroma DB

6.3 Building practical RAG knowledge agents

6.3.1 Everything begins with search and relevance

6.3.2 Building a vector search RAG agent

6.3.3 Building a hybrid search RAG agent

6.4 Adding memory to agents with MCP

6.4.1 Understanding memory form and agent function

6.4.2 Attaching a graph database for memory using MCP

6.4.3 Creating hybrid memory systems with MCP