What will I learn in chapter 6 of AI Agents in Action, Second Edition?

The introduction explains why agents need retrieval for external information and memory for their own past experiences. After reading it you should be able to: Explain why an agent limited to its training data cannot reliably use newly uploaded documents, recent policies, or past conversations. Describe retrieval as finding relevant information outside the model’s training and bringing it into the agent’s context when needed. Explain memory as retrieval of an agent’s own past experiences, enabling it to carry useful context across sessions rather than reset. Explain why an agent can have access to a large external store but can use only a limited amount of it in one LLM call, and how retrieval bridges that gap. Trace the distinction between searching for information using context, typically the user’s input, and adding the retrieved information to an LLM prompt. State that their combination is called retrieval-augmented generation, or RAG. Trace how documents are prepared for RAG and how a query leads to an answer. Distinguish the embedding model’s job from the LLM’s job. Explain why an agent may need retrieved knowledge rather than relying only on an LLM’s training. Describe how semantic search compares a query with stored content by meaning rather than requiring matching keywords, using the section’s dog example to illustrate the difference.

What does section 6.1, Understanding retrieval in AI applications, cover?

The section explains how agents retrieve external knowledge and memory and add relevant pieces to an LLM prompt despite a limited context window. Distinguish external knowledge, such as documents and database tables, from memory that captures experiences, such as past conversations and user preferences. Describe how either can be stored outside the LLM and retrieved for context. Explain why an agent can have access to a large external store but can use only a limited amount of it in one LLM call, and how retrieval bridges that gap. Trace the distinction between searching for information using context, typically the user’s input, and adding the retrieved information to an LLM prompt. State that their combination is called retrieval-augmented generation, or RAG.

What does section 6.1.1, The basics of RAG, cover?

This section explains how RAG retrieves relevant information for an LLM’s answer and how the same retrieval pattern can give agents access to past conversations. Trace how documents are prepared for RAG and how a query leads to an answer. Distinguish the embedding model’s job from the LLM’s job. Explain why an agent may need retrieved knowledge rather than relying only on an LLM’s training. Distinguish vector similarity search from exact lookup and explain how semantic search can match differently worded content.

What does section 6.1.2, Delving into semantic search and document indexing, cover?

Explains how document indexing supports meaning-based retrieval and why semantic search results can be incomplete or unreliable. Explain that document indexing transforms information so it can be recovered, and that the intended way of searching affects the index. Describe how semantic search compares a query with stored content by meaning rather than requiring matching keywords, using the section’s dog example to illustrate the difference. Explain the three retrieval pitfalls described here: a similar result need not be true, fixed top-K results can omit relevant content, and approximate nearest neighbor search can miss the actual closest match. Describe the stated mitigations for top-K omissions.

What does section 6.1.3, Applying vector similarity search, cover?

The section uses TF-IDF to turn documents into sparse vectors, then shows how cosine similarity compares those vectors and where this approach falls short. Distinguish TF-IDF’s word-based measure of importance from semantic embeddings that can represent similar meanings despite different vocabulary. Explain how term frequency and inverse document frequency combine to score a word in a document, using the section’s blue example. Describe how TF-IDF represents each sentence as a vector indexed by the corpus vocabulary.

What does section 6.2, Vector databases and similarity search, cover?

Introduces an in-memory vector database example and explains why matching similar words and phrases is not enough for semantic search. Describe how the example stores vectorized documents for similarity search and what a text query returns. Explain what the demonstrated search matches well and why the section calls for a different way to vectorize documents.

What does section 6.2.1, Demystifying document embeddings, cover?

This section contrasts exact-term retrieval with semantic embeddings, explains how embeddings represent meaning, and shows how to visualize and choose them. Distinguish TF-IDF’s exact-term matching from an embedding’s meaning-based matching, and explain why a retrieval system might combine them. Explain, at a high level, how training leads an embedding network to represent related text as nearby vectors, and distinguish those vectors’ dimensions from TF-IDF dimensions. Describe why the example reduces document embeddings to three dimensions and what the resulting plot illustrates.

What does section 6.3, Building practical RAG knowledge agents, cover?

This section frames vector search as a useful foundation for RAG while noting that practical RAG agents often combine search approaches. Explain that vector search is a strong foundation for RAG, but practical RAG systems and agents often combine it with other search approaches rather than relying on it alone.

What does section 6.3.1, Everything begins with search and relevance, cover?

This section explains why practical RAG agents combine search methods and assess relevance rather than relying on vector similarity alone. Explain why vector search is useful for RAG but cannot by itself determine which documents belong in the final context. Justify combining search techniques according to the data and use case, and describe the section’s example of an agent using multiple search tools.

What does section 6.3.2, Building a vector search RAG agent, cover?

This section builds a movie-script RAG agent using vector search and explains how grounding shapes its answers and exposes a retrieval limitation. Trace how the agent loads a movie script, chunks and embeds it, stores its vectors in Chroma DB, and retrieves them through the search_script tool. Explain why the agent is instructed to answer from retrieved documents rather than fall back on the model’s internal knowledge, and describe a prompt instruction that enforces this. Distinguish weak from strong grounding and explain how the consequences of a wrong answer affect which degree is appropriate.

What does section 6.3.3, Building a hybrid search RAG agent, cover?

This section shows how hybrid RAG combines keyword and vector search, either through rank fusion inside a search tool or through an agent that selects results and cites its sources. Explain why hybrid search combines ranked keyword and vector results rather than their raw scores, and describe how Reciprocal Rank Fusion (RRF) produces a combined ranking. Distinguish a hybrid search tool that internally fuses results from an agent that decides when and how to use separate keyword and vector search functions. Explain why the hybrid RAG agent is instructed to answer from the document knowledge and provide references for each answer.

What does section 6.4, Adding memory to agents with MCP, cover?

This section introduces MCP as a source of tools for agent knowledge and memory systems built with RAG. Explain that RAG can extend an agent beyond its fixed model, while implementing knowledge or memory systems beyond simple cases can be complex, and MCP can provide out-of-the-box tools for them.

What should I know before reading chapter 6?

Agent knowledge and memory (chapter 1): The introduction of agent knowledge and memory as a functional layer.; What is a token? (chapter 2): The idea of tokens as units of model input.; Controlling LLMs with prompt engineering (agent persona) (chapter 2): The use of prompts to provide context to an LLM.; Understanding large language models (chapter 2): Assumes the reader recognizes an LLM as the model used to produce responses.; Core components: Tools, resources, and prompts (chapter 3): Introduces MCP tools as components agents can use.

Chapter 6: Working with memory and knowledge RAG for agents

What you will learn in chapter 6 of AI Agents in Action, Second Edition

The introduction explains why agents need retrieval for external information and memory for their own past experiences.

After this chapter you will be able to

Limits of training data. Explain why an agent limited to its training data cannot reliably use newly uploaded documents, recent policies, or past conversations.
Retrieval of external knowledge. Describe retrieval as finding relevant information outside the model’s training and bringing it into the agent’s context when needed.
Memory of past experiences. Explain memory as retrieval of an agent’s own past experiences, enabling it to carry useful context across sessions rather than reset.
External storage and context limits. Explain why an agent can have access to a large external store but can use only a limited amount of it in one LLM call, and how retrieval bridges that gap.
Retrieval augmentation and RAG. Trace the distinction between searching for information using context, typically the user’s input, and adding the retrieved information to an LLM prompt. State that their combination is called retrieval-augmented generation, or RAG.
RAG ingestion and retrieval. Trace how documents are prepared for RAG and how a query leads to an answer. Distinguish the embedding model’s job from the LLM’s job.
Current knowledge through retrieval. Explain why an agent may need retrieved knowledge rather than relying only on an LLM’s training.
Meaning-based semantic search. Describe how semantic search compares a query with stored content by meaning rather than requiring matching keywords, using the section’s dog example to illustrate the difference.
Limits of semantic retrieval. Explain the three retrieval pitfalls described here: a similar result need not be true, fixed top-K results can omit relevant content, and approximate nearest neighbor search can miss the actual closest match. Describe the stated mitigations for top-K omissions.
TF-IDF and semantic meaning. Distinguish TF-IDF’s word-based measure of importance from semantic embeddings that can represent similar meanings despite different vocabulary.
TF-IDF score calculation. Explain how term frequency and inverse document frequency combine to score a word in a document, using the section’s blue example.
Sparse document vector construction. Describe how TF-IDF represents each sentence as a vector indexed by the corpus vocabulary.
Cosine similarity and limitations. Explain why cosine similarity compares vector direction rather than magnitude, and describe its stated limits with sparse TF-IDF vectors.
Vector Database Search Results. Describe how the example stores vectorized documents for similarity search and what a text query returns.
Word Matching Versus Meaning. Explain what the demonstrated search matches well and why the section calls for a different way to vectorize documents.
Exact Terms and Meaning. Distinguish TF-IDF’s exact-term matching from an embedding’s meaning-based matching, and explain why a retrieval system might combine them.
Meaning in Embedding Vectors. Explain, at a high level, how training leads an embedding network to represent related text as nearby vectors, and distinguish those vectors’ dimensions from TF-IDF dimensions.
Chroma DB semantic retrieval. Explain that Chroma DB is a local vector store suitable for development or small projects, and that document queries can retrieve by semantic meaning rather than only matching keywords.
Cosine distance score interpretation. Explain that the example scores results by cosine distance, calculated as one minus cosine similarity, so zero indicates the most similar items and two indicates semantically opposite items.
Vector Search in Practical RAG. Explain that vector search is a strong foundation for RAG, but practical RAG systems and agents often combine it with other search approaches rather than relying on it alone.
Vector Search and Relevance. Explain why vector search is useful for RAG but cannot by itself determine which documents belong in the final context.
Complementary RAG Search Methods. Justify combining search techniques according to the data and use case, and describe the section’s example of an agent using multiple search tools.
Movie Script Vector Pipeline. Trace how the agent loads a movie script, chunks and embeds it, stores its vectors in Chroma DB, and retrieves them through the search_script tool.
Retrieved Context As Authority. Explain why the agent is instructed to answer from retrieved documents rather than fall back on the model’s internal knowledge, and describe a prompt instruction that enforces this.
Grounding Beyond Prompt Instructions. Explain why grounding also depends on chunk quality, retrieval relevance, and verification, and name a way to check whether answers actually align with retrieved context.
Rank fusion in hybrid search. Explain why hybrid search combines ranked keyword and vector results rather than their raw scores, and describe how Reciprocal Rank Fusion (RRF) produces a combined ranking.
Agent-directed hybrid retrieval. Distinguish a hybrid search tool that internally fuses results from an agent that decides when and how to use separate keyword and vector search functions.
Document-grounded answers with references. Explain why the hybrid RAG agent is instructed to answer from the document knowledge and provide references for each answer.
RAG and MCP for memory. Explain that RAG can extend an agent beyond its fixed model, while implementing knowledge or memory systems beyond simple cases can be complex, and MCP can provide out-of-the-box tools for them.
Architectural pieces of agent memory. Explain how a context window, external storage, state management, and retrieval mechanisms work together to give an agent access to information, and why cognitive memory labels are only loose analogies.
Memory retrieval and updating. Explain how stored memories can augment prompts and why preparing them may differ from preparing knowledge documents.
Graph Memory Structure and Retrieval. Explain how graph memory represents entities, observations, and relationships, and how it retrieves facts by following connections between entities.
MCP Tools for Long-Term Memory. Describe how the agent uses the memory MCP server during a conversation to store and look up facts, relationships, and observations.
Graph Memory Strengths and Limits. Distinguish the relationship queries graph memory handles well from fuzzy questions and identity ambiguities it can struggle with, and explain why the section suggests combining search methods.
Structured use of both memories. Explain that the hybrid memory agent is instructed to use both semantic and graph memory for every interaction, rather than choosing one search type as the earlier hybrid knowledge agent did.
Hybrid memory retrieval flow. Trace how a user statement or query leads from semantic document matching to graph-memory lookup, memory updates, and a response based on the retrieved memories.
Semantic augmentation for retrieval. Explain why episodic and procedural memories can be augmented with semantic meaning, and describe how anticipated user questions help make them retrievable.
Clustered memory compression. Trace how memory or knowledge items are clustered and then summarized into more succinct representations.
Compression versus forgetting. Distinguish when the section recommends compressing similar memories from when forgetting repetitive memories may help.

Cover of AI Agents in Action, Second Edition

liveBook manning.com

Read chapter 6 on liveBook Get the book at manning.com

What it builds on

Agent knowledge and memory (chapter 1) · The introduction of agent knowledge and memory as a functional layer.
What is a token? (chapter 2) · The idea of tokens as units of model input.
Controlling LLMs with prompt engineering (agent persona) (chapter 2) · The use of prompts to provide context to an LLM.
Understanding large language models (chapter 2) · Assumes the reader recognizes an LLM as the model used to produce responses.
Core components: Tools, resources, and prompts (chapter 3) · Introduces MCP tools as components agents can use.

Section by section

Introduction

The introduction explains why agents need retrieval for external information and memory for their own past experiences.

Limits of training data: Explain why an agent limited to its training data cannot reliably use newly uploaded documents, recent policies, or past conversations.
Why it matters: This limitation explains why retrieval and memory are needed.
Retrieval of external knowledge: Describe retrieval as finding relevant information outside the model’s training and bringing it into the agent’s context when needed.
Why it matters: Retrieval lets an agent work with information that was not part of the model’s training.
Memory of past experiences: Explain memory as retrieval of an agent’s own past experiences, enabling it to carry useful context across sessions rather than reset.
Why it matters: Memory helps an agent build on earlier interactions instead of starting over each time.

Key terms: training data, retrieval, context, external knowledge sources, memory, past experiences, user preferences

6.1 Understanding retrieval in AI applications

The section explains how agents retrieve external knowledge and memory and add relevant pieces to an LLM prompt despite a limited context window.

Knowledge and memory sources: Distinguish external knowledge, such as documents and database tables, from memory that captures experiences, such as past conversations and user preferences. Describe how either can be stored outside the LLM and retrieved for context.
Why it matters: The distinction helps clarify what an agent is drawing on when it adds context to a prompt.
External storage and context limits: Explain why an agent can have access to a large external store but can use only a limited amount of it in one LLM call, and how retrieval bridges that gap.
Why it matters: Retrieval lets an agent use a large store without trying to fit the entire store into one prompt.
Retrieval augmentation and RAG: Trace the distinction between searching for information using context, typically the user’s input, and adding the retrieved information to an LLM prompt. State that their combination is called retrieval-augmented generation, or RAG.
Why it matters: Separating search from prompt augmentation makes clear how external information reaches an agent’s LLM call.

Key terms: knowledge, memory, structured, unstructured, context window, external storage, retrieval, augmentation, retrieval-augmented generation, RAG

6.1.1 The basics of RAG

This section explains how RAG retrieves relevant information for an LLM’s answer and how the same retrieval pattern can give agents access to past conversations.

RAG ingestion and retrieval: Trace how documents are prepared for RAG and how a query leads to an answer. Distinguish the embedding model’s job from the LLM’s job.
Why it matters: These two phases explain where an agent’s retrieved answer context comes from.
Current knowledge through retrieval: Explain why an agent may need retrieved knowledge rather than relying only on an LLM’s training.
Why it matters: Retrieval lets an agent address questions whose answers have changed since its model was trained.
Vector and semantic similarity: Distinguish vector similarity search from exact lookup and explain how semantic search can match differently worded content.
Why it matters: Similarity search explains how RAG finds relevant context without requiring identical wording.
Conversation retrieval as memory: Describe how the RAG retrieval pattern applies to an agent’s conversation memory.
Why it matters: The same search pattern can help an agent bring relevant earlier conversation into its current context.

Key terms: RAG, ingestion, retrieval, embedding model, context chunks, augmentation, training cutoff, retrieval-augmented generation, external sources, current context

6.1.2 Delving into semantic search and document indexing

Explains how document indexing supports meaning-based retrieval and why semantic search results can be incomplete or unreliable.

Document indexing for retrieval: Explain that document indexing transforms information so it can be recovered, and that the intended way of searching affects the index.
Why it matters: The way information is prepared affects what a search can recover.
Meaning-based semantic search: Describe how semantic search compares a query with stored content by meaning rather than requiring matching keywords, using the section’s dog example to illustrate the difference.
Why it matters: Searching by meaning can find relevant material without anticipating every keyword or synonym.
Limits of semantic retrieval: Explain the three retrieval pitfalls described here: a similar result need not be true, fixed top-K results can omit relevant content, and approximate nearest neighbor search can miss the actual closest match. Describe the stated mitigations for top-K omissions.
Why it matters: A plausible-looking retrieval result is not proof of correctness or completeness.

Key terms: document indexing, index, semantic search, dense vector embeddings, RAG, memory retrieval, semantic similarity, semantic correctness, top-K, re-ranking

6.1.3 Applying vector similarity search

The section uses TF-IDF to turn documents into sparse vectors, then shows how cosine similarity compares those vectors and where this approach falls short.

TF-IDF and semantic meaning: Distinguish TF-IDF’s word-based measure of importance from semantic embeddings that can represent similar meanings despite different vocabulary.
Why it matters: This distinction explains why matching word-based vectors is not the same as retrieving by meaning.
TF-IDF score calculation: Explain how term frequency and inverse document frequency combine to score a word in a document, using the section’s blue example.
Why it matters: The calculation shows why a word’s score depends on both its document and the wider collection.
Sparse document vector construction: Describe how TF-IDF represents each sentence as a vector indexed by the corpus vocabulary.
Why it matters: Understanding what occupies each vector position makes the subsequent similarity comparison concrete.
Cosine similarity and limitations: Explain why cosine similarity compares vector direction rather than magnitude, and describe its stated limits with sparse TF-IDF vectors.
Why it matters: It explains both why cosine similarity suits document comparison and why a high score does not guarantee a meaningful match.
Pairwise similarity matrix interpretation: Describe how pairwise cosine comparisons form a matrix and how to read a selected document’s row.
Why it matters: The matrix connects the pairwise calculation to the scores shown for a selected document in the example.

Key terms: TF-IDF, semantic embeddings, sparse vector, dense vector, Term Frequency, Inverse Document Frequency, corpus, corpus vocabulary, cosine similarity, cosine distance

6.2 Vector databases and similarity search

Introduces an in-memory vector database example and explains why matching similar words and phrases is not enough for semantic search.

Vector Database Search Results: Describe how the example stores vectorized documents for similarity search and what a text query returns.
Why it matters: This gives you a concrete picture of what the example database stores and returns.
Word Matching Versus Meaning: Explain what the demonstrated search matches well and why the section calls for a different way to vectorize documents.
Why it matters: A high-scoring word match is not necessarily a match in meaning.

Key terms: vector database, similarity searches, similarity score, context, semantic meaning

6.2.1 Demystifying document embeddings

This section contrasts exact-term retrieval with semantic embeddings, explains how embeddings represent meaning, and shows how to visualize and choose them.

Exact Terms and Meaning: Distinguish TF-IDF’s exact-term matching from an embedding’s meaning-based matching, and explain why a retrieval system might combine them.
Why it matters: It helps you recognize when exact keywords and related meanings are each useful for finding documents.
Meaning in Embedding Vectors: Explain, at a high level, how training leads an embedding network to represent related text as nearby vectors, and distinguish those vectors’ dimensions from TF-IDF dimensions.
Why it matters: It gives you a way to interpret what an embedding comparison means without treating each number as a labeled feature.
Three-Dimensional Embedding View: Describe why the example reduces document embeddings to three dimensions and what the resulting plot illustrates.
Why it matters: The plot makes the idea of nearby meanings in vector space easier to see.
Embedding Model Selection: Justify evaluating embedding models against the intended documents and queries rather than choosing one solely for convenience.
Why it matters: Testing with your own material helps reveal whether a generally strong model suits your retrieval task.

Key terms: TF-IDF, semantic embeddings, hybrid retrieval, BM25, embedding networks, dense vectors, semantic structure, TF-IDF dimensions, principal component analysis (PCA), dimensionality reduction

6.2.2 Querying document embeddings from Chroma DB

This section uses Chroma DB to illustrate semantic document retrieval and explains how to interpret results scored by cosine distance.

Chroma DB semantic retrieval: Explain that Chroma DB is a local vector store suitable for development or small projects, and that document queries can retrieve by semantic meaning rather than only matching keywords.
Why it matters: This is the retrieval capability the later RAG agent examples build on.
Cosine distance score interpretation: Explain that the example scores results by cosine distance, calculated as one minus cosine similarity, so zero indicates the most similar items and two indicates semantically opposite items.
Why it matters: Reading the score in the wrong direction would invert which results appear most similar.

Key terms: Chroma DB, local vector store, semantic meaning, cosine distance, cosine similarity

6.3 Building practical RAG knowledge agents

This section frames vector search as a useful foundation for RAG while noting that practical RAG agents often combine search approaches.

Vector Search in Practical RAG: Explain that vector search is a strong foundation for RAG, but practical RAG systems and agents often combine it with other search approaches rather than relying on it alone.
Why it matters: This distinction sets up why a practical RAG agent may need more than vector search.

Key terms: similarity or vector search, RAG systems, RAG agents, combination of search approaches

6.3.1 Everything begins with search and relevance

This section explains why practical RAG agents combine search methods and assess relevance rather than relying on vector similarity alone.

Vector Search and Relevance: Explain why vector search is useful for RAG but cannot by itself determine which documents belong in the final context.
Why it matters: Finding similar documents is not the same as choosing useful context for an answer.
Complementary RAG Search Methods: Justify combining search techniques according to the data and use case, and describe the section’s example of an agent using multiple search tools.
Why it matters: The right mix of search tools depends on what information the agent must find.

Key terms: vector/similarity search, relevant information, RAG context, supplementary technique, vector/hybrid search, relational database search, RAG agent

6.3.2 Building a vector search RAG agent

This section builds a movie-script RAG agent using vector search and explains how grounding shapes its answers and exposes a retrieval limitation.

Movie Script Vector Pipeline: Trace how the agent loads a movie script, chunks and embeds it, stores its vectors in Chroma DB, and retrieves them through the search_script tool.
Why it matters: This sequence shows where the agent’s retrievable knowledge comes from.
Retrieved Context As Authority: Explain why the agent is instructed to answer from retrieved documents rather than fall back on the model’s internal knowledge, and describe a prompt instruction that enforces this.
Why it matters: A plausible answer is not necessarily supported by the script the agent was asked to use.
Grounding Strength And Stakes: Distinguish weak from strong grounding and explain how the consequences of a wrong answer affect which degree is appropriate.
Why it matters: The required evidence for an answer should match the risk of getting it wrong.
Grounding Beyond Prompt Instructions: Explain why grounding also depends on chunk quality, retrieval relevance, and verification, and name a way to check whether answers actually align with retrieved context.
Why it matters: Instructions express the desired behavior; checking retrieved evidence and outputs shows whether it happened.
Vector Search Time Limitation: Describe what the two script questions reveal about this vector-only agent’s retrieval and grounded answering.
Why it matters: A missing answer may reveal a retrieval limitation rather than a failure to follow grounding instructions.

Key terms: semantic chunks, OpenAI embedding model, Chroma DB, search_script, vector/similarity search, grounding, retrieved context, hallucination, weak grounding, strong grounding

6.3.3 Building a hybrid search RAG agent

This section shows how hybrid RAG combines keyword and vector search, either through rank fusion inside a search tool or through an agent that selects results and cites its sources.

Rank fusion in hybrid search: Explain why hybrid search combines ranked keyword and vector results rather than their raw scores, and describe how Reciprocal Rank Fusion (RRF) produces a combined ranking.
Why it matters: Rank fusion lets hybrid search use both kinds of results without depending on their incompatible score scales.
Agent-directed hybrid retrieval: Distinguish a hybrid search tool that internally fuses results from an agent that decides when and how to use separate keyword and vector search functions.
Why it matters: It separates what the search tool does automatically from the relevance decisions entrusted to the agent.
Document-grounded answers with references: Explain why the hybrid RAG agent is instructed to answer from the document knowledge and provide references for each answer.
Why it matters: A cited answer can be checked against the supplied documents rather than accepted because it sounds familiar.

Key terms: hybrid search, keyword search, vector search, Reciprocal Rank Fusion (RRF), ranked list, hybrid search tool, vector search function, keyword search function, relevance, grounding

6.4 Adding memory to agents with MCP

This section introduces MCP as a source of tools for agent knowledge and memory systems built with RAG.

RAG and MCP for memory: Explain that RAG can extend an agent beyond its fixed model, while implementing knowledge or memory systems beyond simple cases can be complex, and MCP can provide out-of-the-box tools for them.
Why it matters: It explains why MCP is relevant when an agent needs knowledge or memory beyond its fixed model.

Key terms: RAG, MCP, knowledge, memory, out-of-the-box tools

6.4.1 Understanding memory form and agent function

The section explains agent memory through the storage, working state, and retrieval mechanisms that make information available to a prompt, using cognitive memory terms only as shorthand.

Architectural pieces of agent memory: Explain how a context window, external storage, state management, and retrieval mechanisms work together to give an agent access to information, and why cognitive memory labels are only loose analogies.
Why it matters: Thinking in terms of actual components makes it clearer what an agent can keep in a prompt and what it must store and retrieve.
Multimodal sensory retrieval: Describe what the section means by sensory memory and how its usefulness and reliability compare with text retrieval.
Why it matters: The label should not suggest a separate memory faculty or that image and audio retrieval are as dependable as text retrieval.
Memory retrieval and updating: Explain how stored memories can augment prompts and why preparing them may differ from preparing knowledge documents.
Why it matters: It connects persistent memory to what the agent actually sees in its current prompt without treating every memory like a full document.
Storage matched to memory content: Distinguish storage choices for facts and statistics from those for relationships, using the examples in the section.
Why it matters: Choosing storage by the information being remembered is more useful than assuming one memory store fits every need.

Key terms: context window, external storage, state management, retrieval mechanisms, short-term memory, long-term memory, sensory memory, multimodal retrieval, embed-and-search, modality-specific index

6.4.2 Attaching a graph database for memory using MCP

This section shows how an agent uses an MCP-connected graph database as long-term memory, and why graph memory alone may need a complementary search method.

Graph Memory Structure and Retrieval: Explain how graph memory represents entities, observations, and relationships, and how it retrieves facts by following connections between entities.
Why it matters: This explains why graph memory is useful when an agent needs to recall how things are connected.
MCP Tools for Long-Term Memory: Describe how the agent uses the memory MCP server during a conversation to store and look up facts, relationships, and observations.
Why it matters: The MCP tools connect conversational decisions to persistent graph memory.
Graph Memory Strengths and Limits: Distinguish the relationship queries graph memory handles well from fuzzy questions and identity ambiguities it can struggle with, and explain why the section suggests combining search methods.
Why it matters: Recognizing these limits explains why an agent may need more than graph lookup for reliable recall.

Key terms: graph database, entities, nodes, edges, observations, relationship traversal, memory MCP server, add_observation, add_relationship, query_facts

6.4.3 Creating hybrid memory systems with MCP

This section explains how an agent combines semantic and graph memory through MCP, then distinguishes the learning example from production-ready memory.

Structured use of both memories: Explain that the hybrid memory agent is instructed to use both semantic and graph memory for every interaction, rather than choosing one search type as the earlier hybrid knowledge agent did.
Why it matters: Using both stores is what makes this a hybrid memory pattern rather than a choice between search methods.
Hybrid memory retrieval flow: Trace how a user statement or query leads from semantic document matching to graph-memory lookup, memory updates, and a response based on the retrieved memories.
Why it matters: The order shows how a meaning-based match can lead the agent to related graph memories before it answers.
Example and production memory limits: Distinguish the ephemeral teaching examples from production memory, and describe where the substantive memory work lies beyond the MCP wrapper.
Why it matters: A working demonstration does not by itself provide durable or reliable memory.

Key terms: MCP servers, graph memory, semantic memory, agent instructions, hybrid memory agent, semantic store, entities, observations, ephemeral, MCP server

6.4.4 Semantic augmented memory and applications to semantic, episodic, and procedural memory

The section distinguishes three forms of memory and explains how semantic augmentation makes event and process memories easier to retrieve.

Three forms of memory: Distinguish episodic, procedural, and semantic memory by the information each represents.
Why it matters: These distinctions clarify what kind of memory an agent might need to store and retrieve.
Semantic augmentation for retrieval: Explain why episodic and procedural memories can be augmented with semantic meaning, and describe how anticipated user questions help make them retrievable.
Why it matters: Augmentation helps connect a stored experience or step to the way someone might later ask for it.

Key terms: episodic memory, procedural memory, semantic memory, relational database search/retrieval pattern, semantic/vector storage, semantic memory augmentation

6.4.5 Uncluttering memory with compression and forgetting

This section explains how clustering and summarizing can reduce memory and knowledge clutter, when compression or forgetting is useful, and why production forgetting needs care.

Clustered memory compression: Trace how memory or knowledge items are clustered and then summarized into more succinct representations.
Why it matters: Grouping similar items lets an agent retain significant information without keeping every item separately.
Compression versus forgetting: Distinguish when the section recommends compressing similar memories from when forgetting repetitive memories may help.
Why it matters: The right way to reduce clutter depends on whether information should be condensed or removed.
Knowledge compression timing: Describe how the usefulness and frequency of compression differ between knowledge sources and memory, and what multiple passes can provide.
Why it matters: Compression is more useful when its timing and degree fit the material being retrieved.
Production forgetting constraints: Explain why deciding what a production agent forgets involves more than clearing stale facts, especially in regulated settings.
Why it matters: A cleaner memory store is not an improvement if it loses information the system must retain.

Key terms: memory compression, semantic augmentation, clustered, compression function, forgetting, repetitive memories, knowledge compression, periodic compression, compression levels, production memory

6.5 Exercises

The section gives five hands-on exercises for comparing retrieval methods, persisting vectors, and testing vector-only and hybrid RAG agents.

6.6 Summary

The section summarizes how agents retrieve external knowledge, combine search methods, and manage memory to produce grounded, scalable answers.

Where this is used later

Knowledge and memory (chapter 11) · document indexing for retrieval
The memory module and the MCP memory server (chapter 10) · The distinction between vector-based similarity search and other ways of organizing knowledge.
Tips for building a customer support agent (chapter 11) · Using retrieval to supply relevant knowledge to an agent.
Summary (chapter 11) · Introduces RAG knowledge agents.
Introducing agent evaluation and feedback (chapter 7) · A RAG agent whose retrieved context can serve as a source for grounding.
Exploring TDAD in practice (chapter 7) · The idea of a RAG agent that searches knowledge to answer questions.
Grounding the RAG agent (chapter 7) · Grounding the RAG agent depends on treating retrieved material as the answer’s source.
Coding and testing the RAG agent (chapter 7) · Provides context for coding and testing a RAG agent.

Concepts explained in this chapter

similarity searchRetrieval is the process of finding relevant information from an external source and bringing it into a model's context.
embeddingA dense vector is a numerical vector in which the representation generally assigns a value to every dimension.
vectorizationVectorization is the transformation of documents or other data into numerical vector representations.
hybrid searchHybrid search combines multiple retrieval methods, typically including vector-based and lexical or structured search, to improve information retrieval.
term frequency-inverse document frequencyTF-IDF is a term-weighting technique that scores a word by its frequency in a document relative to its frequency across a collection of documents.
sparse vectorA sparse vector is a vector in which most positions contain zero or otherwise absent values.
vector databaseA vector database server is a service that exposes storage and similarity retrieval over vector representations.
cosine similarityCosine similarity measures the similarity of two nonzero vectors by the cosine of the angle between them, independently of their magnitudes.
memoryAgent memory is information retained from prior interactions or external sources that can augment an agent's context.
agentic RAGAgentic RAG is a retrieval-augmented generation approach in which an agent dynamically decides how and when to retrieve information, often iterating or using tools before generating a response.
text chunkSemantic chunking is the segmentation of source material into sections that preserve meaningful units of content.
retrieval-augmented generationRetrieval-augmented generation is a technique that retrieves pertinent information at generation or decision time and adds it to a model's context.
cosine distanceCosine distance is a metric equal to one minus cosine similarity that measures angular dissimilarity between vectors.
Chroma DBChroma DB is a database system for storing and retrieving vector embeddings and associated document data.
inverse document frequencyInverse document frequency is a logarithmic measure of how uncommon a term is across a document collection.
term frequencyTerm frequency is the frequency of a term in a document, commonly normalized by the document's total number of words.
semantic memorySemantic memory is memory retrieved according to the meaning of stored information rather than only its explicit graph relationships or literal terms.
groundingGrounding is the process of anchoring a model's responses in relevant external information, data, or context.
hallucinationA hallucination is model-generated content that is plausible-sounding but wrong or contradictory to the relevant source information.
lexical searchKeyword matching retrieves content by comparing literal terms in a query with terms in stored content.
prompt augmentationPrompt augmentation is the addition of retrieved information to an input prompt to provide a model with relevant context.
sensory memorySensory memory in AI is multimodal retrieval of information from modality-specific indexes such as image or audio stores.
working memoryShort-term memory is the recent interaction history of the current session that remains directly in the model's context window.
compressionCompression is the reduction or summarization of stored information while preserving information needed for later retrieval or use.
reciprocal rank fusionReciprocal rank fusion combines ranked result lists by summing the reciprocal of each result's rank offset across the lists.
hybrid memory systemA hybrid memory system combines semantic retrieval with graph-based retrieval and updates to support an agent's memory.
domain knowledgeDomain knowledge is specialized understanding of the concepts, facts, practices, relationships, and constraints of a particular subject area.
context windowThe finite amount of input and conversational context that an agent can retain or process in one interaction.
knowledge and memoryKnowledge and memory are the information resources an agent maintains or accesses to support decisions across interactions and tasks.
knowledge graphA knowledge graph is a graph-structured representation of entities, entity types, observations, and relationships among them.
state persistencePersistence is the continuation of a system's goal-directed operation across multiple steps or interactions.
MCPThe Model Context Protocol is an open protocol for connecting AI applications and models with external tools, data sources, and contextual resources.
pipelineIngestion is the phase of preparing source documents by loading, transforming, embedding, and storing them for later retrieval.
tool chainingTool chaining is the process in which the output of one tool call becomes the input to a subsequent tool call.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.