6 Working with memory and knowledge RAG for agents

Now that we’ve explored how reasoning and planning can create smarter agents, we will look at how retrieval and context augmentation can be used to support knowledge and memory in agents. We start with a look at the basis for retrieval and how we can search by semantic meaning using dense vector embeddings. Then we will look at the practical side of using retrieval to build knowledge agents, agents that have access to external resources to help answer questions. Finally, we will employ retrieval to find relevant previous experiences in the form of memory that will be used to help augment agent context.

6.1 Understanding retrieval in AI applications

Retrieval in agent and chat applications is a mechanism for obtaining knowledge/memories stored in an external, long-lived storage system. Unstructured knowledge includes conversation or task histories, facts, preferences, or other items necessary for contextualizing a prompt. Structured knowledge, normally stored in databases or files, is accessed through native functions or plugins. Memories may consist of the previous conversation thread, previous agent experiences, previous facts or preferences about the user, or other experiential information.

6.1.1 The basics of retrieval augmented generation (RAG)

6.1.2 Delving into semantic search and document indexing

6.1.3 Applying vector similarity search

6.2 Vector databases and similarity search

6.2.1 Demystifying document embeddings

6.2.2 Querying document embeddings from Chroma

6.3 Building practical RAG knowledge agents

6.3.1 Everything begins with search and relevance

6.3.2 Building a vector search RAG agent

6.3.3 Building a hybrid search RAG agent

6.4 Adding memory to agents with MCP

6.4.1 Understanding memory form and agent function

6.4.2 Implementing a graph database for memory using MCP

6.4.3 Creating hybrid memory systems with MCP

6.4.4 Semantic augmented memory and applications to semantic, episodic, and procedural memory

6.4.5 Uncluttering memory with compression and forgetting

6.5 Exercises

6.6 Summary