Chapter 6: Adding memory to your agent

Introduction

The chapter introduces why agents need memory and defines the context-growth problem that memory management must address.

• Memory and agent continuity: Explain why memory matters for an LLM agent: without it, the agent cannot recall earlier events in a task, continue a conversation from a previous session, or learn from experience.
  Why it matters: Memory is what lets an agent carry useful context forward instead of treating every interaction as new.
• Context explosion: Describe context explosion as context taking up a large share of the model’s context window or growing past it during complex problem solving.
  Why it matters: This names the context-growth problem that the chapter’s management strategies are intended to prevent.

Key terms: memory, stateless tool, context explosion, context window

6.1 The anatomy of agent memory

The section introduces why memory matters to an agent, while previewing the memory distinctions explained later.

• Memory and agent continuity: Explain why memory makes an agent more useful across interactions and what happens when an agent lacks it.
  Why it matters: Memory is what allows an agent to use what happened before rather than treating every interaction as new.

Key terms: memory, previous conversations, experiences

6.1.1 Limitations of the current memory architecture

This section explains why the agent’s raw ExecutionContext fails to manage information during long tasks and across sessions.

• Current memory architecture limits: Explain how the raw ExecutionContext handles information and why that causes both a context-space problem during execution and a forgetting problem between sessions.
  Why it matters: These two failure modes explain why the agent needs memory management.

Key terms: ExecutionContext, context, window limit, memory management

6.1.2 Three challenges of the current memory architecture

This section defines sessions and short- and long-term memory, then explains three limits of the current agent architecture: growing context, broken conversation continuity, and knowledge that does not persist across sessions.

• Session and memory boundaries: Distinguish a session, short-term memory, and long-term memory, including what each retains and how information enters the two memory stores.
  Why it matters: These boundaries explain why preserving a conversation and remembering something for a future conversation are different problems.
• Context growth during execution: Explain why the current memory approach can become problematic during a complex execution and identify the challenge it creates.
  Why it matters: Memory management must address growth within a run, not just persistence between runs.
• Conversation continuity across runs: Explain why separate run() calls do not inherently maintain one conversation and describe what must be preserved to support follow-up requests.
  Why it matters: A follow-up question or resumed task only makes sense if the agent can recover the preceding conversation.
• Knowledge persistence across sessions: Distinguish persistent knowledge from within-session continuity and describe how useful information can reach a later, separate session.
  Why it matters: A personalized agent needs selected facts to remain available even when a user starts a new conversation.

Key terms: session, short-term memory, long-term memory, ExecutionContext.events, context explosion, tool calls, compress context, conversation continuity, session management, session_id

6.1.3 Context engineering and memory

The section connects three memory patterns to context-engineering strategies and distinguishes managing context within a run, across a session, and after a session.

• Three memory time scales: Distinguish in-run context management, continuous execution, and long-term memory by when each must preserve or manage information.
  Why it matters: The time span of the information determines which memory pattern addresses the problem.
• Memory and context strategies: Explain how the patterns relate to reduce, write, and retrieval, and describe the reduction methods named in this section.
  Why it matters: Memory is also about selecting what reaches the LLM now, not just saving everything.
• Current context storage locations: Distinguish where the events come from when managing context within one run versus across calls in a session.
  Why it matters: The same current-context problem can arise from events in an active run or from a session's stored history.

Key terms: in-run context management, continuous execution, long-term memory, run(), session, reduce, write, retrieval, sliding window, compaction

6.2 Managing context during execution

6.2.1 Separating storage from presentation

This section explains how to preserve an agent’s complete execution history while tailoring each LLM request through callbacks.

• Stored history and LLM presentation: Explain why context management keeps ExecutionContext.events intact while processing only LlmRequest.contents for each LLM call.
  Why it matters: Keeping the original record lets you optimize what the model sees without losing evidence or breaking references to earlier results.
• Before-LLM callback behavior: Describe how before_llm_callback processes a request just before an LLM call and what its two possible return values do.
  Why it matters: The callback is the point where an agent can tailor an LLM call without rewriting its execution record.
• Threshold-controlled optimizer callbacks: Describe how create_optimizer_callback uses an optimization function and token threshold, and where an optimizer records execution state.
  Why it matters: A configurable threshold lets the agent apply request optimization only when it is needed.

Key terms: ExecutionContext.events, LlmRequest.contents, ground truth, computed projection, before_llm_callback, ExecutionContext, LlmRequest, LlmResponse, create_optimizer_callback, closure pattern

6.2.2 Sliding-window strategy

This section explains how a sliding window limits an agent’s context by preserving the original question and retaining only recent subsequent items, then shows why message-count limits are imperfect.

• Sliding-window message selection: Describe how the sliding-window strategy finds the original user question, preserves everything through that message, and keeps only the most recent window_size subsequent items.
  Why it matters: This explains how the agent can shorten its context without dropping the question it is trying to answer.
• Message-count window limitations: Explain the tradeoff in choosing a small or large message window and why counting messages does not reliably limit context size.
  Why it matters: A predictable message window is not necessarily a reliable limit on what the agent sends to the model.

Key terms: sliding-window strategy, window_size, original question, context-explosion problem, message count, token counts

6.2.3 Token counting

This section explains how to measure an agent’s context in tokens and why to keep it focused even when it fits within the model’s context window.

• Token-Based Context Limits: Explain why message count is not a reliable measure of context size and why context management must measure tokens instead.
  Why it matters: Counting messages can hide a single result that uses a large share of the available context.
• Full Request Token Counting: Describe how to count the tokens in an LlmRequest by using build_messages to obtain the API-bound message format, then applying tiktoken, including what the count covers and how its accuracy differs by model.
  Why it matters: Counting the whole formatted request gives the agent a more useful measure than counting only conversation text.
• Below-Limit Context Optimization: Justify minimizing context to what the task requires even when the prompt fits within the model’s advertised context window.
  Why it matters: Staying under the limit does not by itself make a prompt efficient or effective.

Key terms: token count, message count, context management, tiktoken, LlmRequest, build_messages, tool definitions, context bloat, context window, lost in the middle

6.2.4 Compaction strategy

This section explains how compaction keeps an agent’s context lean by replacing processed tool data with references, and when its costs justify using it.

• On-demand context through compaction: Explain how compaction replaces already processed data with lightweight references so the agent can request the data again if needed.
  Why it matters: This is the reason compaction can keep prompts small without making previously accessed data permanently unavailable.
• Tool call and result targets: Distinguish compaction of write-operation ToolCall arguments from compaction of read-operation ToolResult content.
  Why it matters: The location of the large data determines whether the tool call or its result should be compacted.
• Call before result processing: Describe how the implementation processes ToolCalls and ToolResults in one pass, retaining call arguments for later result compaction and using separate rules for each target.
  Why it matters: The ordering of calls and results makes the section’s single-pass implementation possible.
• Refetching cost tradeoff: Explain when replacing large data with a reference is worthwhile despite the possibility of another tool call.
  Why it matters: Saving context is not free if the agent later needs to retrieve the data again.
• Prompt cache preservation threshold: Explain why changing older messages can undermine prompt caching and why compaction should wait until its benefit outweighs cache misses.
  Why it matters: Compacting too early can sacrifice a caching benefit that would have made an unchanged prompt cheaper.

Key terms: compaction, scope by default, lightweight references, ToolCall, ToolResult, write operations, read operations, TOOLCALL_COMPACTION_RULES, TOOLRESULT_COMPACTION_RULES, LLM tokens

6.2.5 Summarization strategy

Explains how to summarize older parts of an agent’s conversation when compaction is insufficient, while preserving the task and recent messages.

• Summarization Targets and Purpose: Explain when summarization is needed and which messages it should replace. Describe why the original user question remains intact.
  Why it matters: The agent needs a shorter context without losing sight of the user’s task.
• Summarized Context Reconstruction: Trace how the strategy rebuilds the context from the user question, a summary of intermediate messages, and recent messages. State how it avoids summarizing the same portion again.
  Why it matters: The reconstructed context keeps the task and latest exchanges available without repeatedly processing old material.
• Compaction and Summarization Tradeoffs: Distinguish compaction from summarization in how each chooses information to remove and whether that information can be recovered. Justify why compaction comes first.
  Why it matters: Choosing the reversible, predictable option first avoids unnecessary loss of the agent’s reasoning.

Key terms: summarization, intermediate process, user’s original question, keep_recent, summary message, context.state, last summarization position, compaction, deterministic, semantic

6.2.6 Hierarchical context management

This section presents a cost-aware order for managing context size and shows how to run those checks through an agent callback.

• Hierarchical context strategy: Explain when to leave context unchanged, compact it, or summarize it, and why the strategies are applied in that order.
  Why it matters: The order keeps context within limits without paying for summarization when a cheaper step will do.
• Callback-based strategy integration: Describe how ContextOptimizer runs the context-management stages before an LLM call and stops once the token count falls below the threshold.
  Why it matters: Rechecking after each stage lets the agent stop optimizing as soon as the context fits.

Key terms: token count, threshold, compaction, summarization, prompt-caching benefits, ContextOptimizer, __call__, before_llm_callback

6.3 Continuous execution: Session and state management

This section explains why sessions are needed to preserve conversations across agent runs and how saved execution state allows tool approval without blocking the agent.

• Conversation continuity across runs: Explain why the current agent forgets conversations between run() calls and how a Session lets a later call continue the same conversation.
  Why it matters: A session gives separate agent runs access to the conversation they need to continue.
• Saved state for delayed approval: Describe why terminal-based tool approval is unsuitable for production services and how saving execution state in a Session lets the agent pause and resume after approval arrives.
  Why it matters: Saved state allows approval to arrive later without keeping a production service waiting at a terminal prompt.

Key terms: ExecutionContext, run(), Session, session_id, intermediate execution state, HITL, input(), tool approval

6.3.1 The Session class

This section defines a Session as persistent storage across agent runs, distinguishes it from a single-run ExecutionContext, and explains its fields.

• Session and ExecutionContext lifecycles: Explain why a session persists across run() calls while an ExecutionContext lasts for one run, and describe how information moves between them.
  Why it matters: This distinction explains how an agent can continue a conversation or resume an interrupted task across runs.
• Session fields and flexible state: Describe what the Session class’s main fields store and why its state field is generic.
  Why it matters: The field design lets one session structure retain both conversation events and the state needed by different kinds of tasks.

Key terms: session, ExecutionContext, run(), conversation history, execution state, Session class, session_id, user_id, events, state

6.3.2 Managing sessions with SessionManager

This section separates session data from session management, defines a shared asynchronous manager interface, and shows how an in-memory implementation behaves.

• Session manager responsibilities: Explain why Session needs a separate manager and describe what creating, getting, getting or creating, and saving a session do.
  Why it matters: Keeping data and storage responsibilities separate lets the agent manage sessions consistently.
• Shared manager interface: Describe how the abstract SessionManager interface relates to storage-specific implementations and why get_or_create can be a regular method.
  Why it matters: A shared interface lets storage implementations differ without changing how callers use the manager.
• Asynchronous interface rationale: Justify why the manager interface declares async methods even though an in-memory manager does not need asynchronous storage operations.
  Why it matters: Designing for storage that may require async operations avoids a caller-side rewrite later.
• In-memory session behavior: Describe where InMemorySessionManager stores sessions, what its create, get, and save methods do, and its limits.
  Why it matters: These behaviors determine when a session can be reused and when a more durable store is needed.

Key terms: Session, SessionManager, session_id, get_or_create, Abstract class, get, create, async, asynchronous interface, InMemorySessionManager

6.3.3 Integrating sessions into the agent

This section integrates sessions into the agent so conversation history and runtime state can carry across executions.

• Session-backed execution context: Explain how ExecutionContext now refers to a Session for events and state while preserving the existing context.events and context.state interface.
  Why it matters: This lets the agent retain conversation state without requiring existing context-using code to change.
• Agent session execution lifecycle: Trace how the agent selects a SessionManager and how run() retrieves or creates, uses, and saves a session.
  Why it matters: Loading and saving the same session allows successive agent runs to share conversation history.

Key terms: ExecutionContext, Session, events, state, properties, backward compatibility, SessionManager, InMemorySessionManager, session_id, get_or_create()

6.3.4 Basic example: Multiturn conversation

Shows how reusing a session ID lets an agent retain conversational context across run() calls, while omitting or changing it keeps calls independent.

• Shared session context flow: Trace how two run() calls with the same session_id share conversational context through the Session, its events, and ExecutionContext.
  Why it matters: This explains why the agent can recall something a user said in an earlier call.
• Independent calls and session choice: Distinguish the behavior of calls with no session_id or different IDs from calls that reuse one, and describe when independent versus continuing conversations are useful.
  Why it matters: Choosing whether to reuse a session ID determines whether later messages can draw on earlier ones.

Key terms: session_id, Session, session.events, ExecutionContext, context.events, one-off tasks, GAIA benchmark problems

6.3.5 Data structures for tool confirmation

This section defines the data structures and session-state serialization needed to represent tool calls awaiting human approval.

• Approval request and decision types: Distinguish PendingToolCall, which describes a tool call awaiting approval, from ToolConfirmation, which records the user's decision. Describe the fields that preserve the original call, present the request, match a decision to a call, and optionally change its arguments.
  Why it matters: These types let the agent present a specific proposed action and associate the user's decision with it.
• Agent result approval status: Explain how AgentResult distinguishes a completed execution from one paused for approval or one that errored, and state what it includes when approval is pending.
  Why it matters: A caller needs to know whether it has a final result or must request approval for pending actions.
• Session state serialization pattern: Describe how pending tool calls are converted for storage in session state and reconstructed on a later run, and explain why that conversion is needed.
  Why it matters: Storing a restorable representation lets an approval workflow continue across runs rather than losing its pending call.

Key terms: PendingToolCall, ToolCall, confirmation_message, ToolConfirmation, tool_call_id, approved, AgentResult, status, complete, pending_confirmation

6.3.6 Extending tools for confirmation

This subsection shows how tools declare whether they need user approval and how they produce an approval message before execution.

• Tool-level approval requirements: Explain how confirmation requirements are attached to tools and checked before execution. Describe how the updated @tool decorator marks a tool such as delete_file for approval while get_weather executes without it.
  Why it matters: Putting the requirement on the tool lets the agent distinguish actions that need approval from those that can run immediately.
• Approval message construction: Describe how a tool builds the message shown when approval is requested, including how tool-call details can be inserted and where the resulting message is held during a pause.
  Why it matters: A message containing the proposed action’s details helps the user decide whether to approve it.

Key terms: BaseTool, requires_confirmation, @tool decorator, confirmation_message_template, get_confirmation_message(), PendingToolCall, {name}, {arguments}, {arguments[key]}

6.3.7 Implementing pause and resume in the agent

This section implements human approval of tool calls by pausing the agent before protected tools run, then resuming with the user's decisions visible to the LLM.

• Internal confirmation handling: Justify why tool confirmation belongs in the agent's internal request-preparation logic rather than a user callback.
  Why it matters: Putting confirmation in the right layer makes approval results available to the LLM at the right time.
• Pre-execution tool gating: Describe how act() handles a tool call that requires confirmation compared with one that does not.
  Why it matters: The check prevents a protected action from taking effect before a person can approve it.
• Run loop pause and resume: Trace how run() pauses when calls are pending and resumes when the user supplies approval decisions.
  Why it matters: This is how the agent gives a person time to decide without losing the pending work.
• Approval results before requests: Explain the order in which _prepare_llm_request() processes decisions and constructs the next LLM request.
  Why it matters: The LLM needs the approval outcome in its next request to continue appropriately.
• Selective approval processing: Describe how _process_confirmations() treats approved, rejected, omitted, and modified pending calls.
  Why it matters: Selective approval lets a person permit safe calls, alter their arguments, and give the LLM clear feedback about blocked calls.

Key terms: _prepare_llm_request(), before_llm_callback, ToolResults, act(), requires_confirmation, PendingToolCall, context.state, ToolResult, run(), step()

6.3.8 Complete example: HITL workflow

This section demonstrates a complete human-in-the-loop file-deletion workflow, from pausing for approval to resuming the same session.

• File deletion approval workflow: Trace how a confirmation-required delete_file call pauses, exposes a pending approval request, and resumes after the user approves it.
  Why it matters: The example shows how to require a person’s approval before an agent performs a consequential action.

Key terms: HITL workflow, requires_confirmation, pending_confirmation, pending_tool_calls, ToolConfirmation, session_id

6.4 Long-term memory: Accumulating knowledge across sessions

This section explains how long-term memory carries useful information across sessions by storing it externally and retrieving it for later context.

• Memory across session boundaries: Explain why session continuity is not enough and how long-term memory makes information available in later sessions.
  Why it matters: This distinction explains how an agent can use knowledge from a past session rather than starting over.

Key terms: Session, SessionManager, session_id, long-term memory, write (storage), retrieval (search)

6.4.1 The structure of long-term memory

This section distinguishes session continuity from long-term memory and explains how selected information is stored, retrieved, and managed across sessions.

• Session and memory roles: Distinguish SessionManager’s preservation of conversation content within one session from MemoryManager’s preservation of useful information for future sessions.
  Why it matters: It clarifies why keeping a conversation going and remembering something for a later conversation need different components.
• Selective storage and retrieval: Explain the two operations of long-term memory and why extracting information worth preserving is preferable to storing whole conversations.
  Why it matters: Selecting useful information makes later retrieval more valuable than simply retaining everything said.
• Memory conflict operations: Describe ADD, UPDATE, DELETE, and SKIP, and justify why the chapter’s problem-solving records use only ADD and SKIP.
  Why it matters: The choice to retain distinct attempts, including failures, explains the chapter’s simpler memory design.

Key terms: SessionManager, MemoryManager, session_id, horizontal continuity, vertical continuity, storage (Write), retrieval, vector embeddings, information extraction, ADD

6.4.2 Information extraction: Structured output

This section shows how to extract structured problem-solving memories from an agent’s execution history, prepare them for retrieval, and check for duplicates before storage.

• Structured memory purpose: Explain why a simple text fact can be stored as is, while a GAIA problem-solving record needs to capture the problem, approach, and result.
  Why it matters: The memory format determines whether a later attempt can make useful use of an earlier one.
• Task memory schema: Describe the fields in the Pydantic TaskMemory schema and distinguish the text used to find similar problems from information used after retrieval.
  Why it matters: The agent can search by the problem while retaining the attempted solution and outcome for later use.
• Structured extraction from history: Trace how the LLM turns an agent’s execution history into a structured memory, including the prompt’s requested information.
  Why it matters: A useful memory is synthesized from the whole attempt rather than from the final answer alone.
• Simple LLM extraction interface: Distinguish the prompt-based ask() method from the agent-execution generate() method and state how ask() requests formatted output.
  Why it matters: Information extraction can request formatted output through a simpler call than the agent-execution interface.
• Duplicate memory check: Explain how and why a proposed memory is checked against existing memories before storage.
  Why it matters: Checking before storage keeps the memory collection from accumulating repeated accounts of one problem.

Key terms: structured memory, GAIA problem-solving records, Pydantic, TaskMemory, task_summary, approach, final_answer, is_correct, extraction prompt, execution history

6.4.3 Building a vector store with ChromaDB

Introduces ChromaDB collections and shows how they store and retrieve documents through automatic embedding and similarity search.

• ChromaDB collections and contents: Describe a ChromaDB collection as a group of related documents that stores document text, embedding vectors, and metadata together.
  Why it matters: Collections provide the structure in which the agent’s searchable information is kept.
• Automatic embedding and retrieval flow: Trace the basic ChromaDB flow from creating an in-memory client and collection to adding document text and querying by text when an embedding function is specified.
  Why it matters: This is the basic path for storing text and finding similar documents without implementing the vector calculations yourself.
• Metadata alongside searchable documents: Explain what the metadatas parameter stores and how that information can be used in search results.
  Why it matters: Metadata lets a search result carry useful context beyond the document text.

Key terms: collection, document text, embedding vectors, metadata, chromadb.Client(), get_or_create_collection(), embedding_function, add(), query_texts, query()

6.4.4 Implementing TaskMemoryManager

This section implements a TaskMemoryManager that turns agent execution history into structured GAIA task memories, avoids duplicates, stores and searches them in ChromaDB, and makes the manager available to agent tools.

• Structured memory extraction: Describe how TaskMemoryManager prepares execution history for the LLM and obtains a TaskMemory object from it.
  Why it matters: Structured extraction makes an execution history usable as a stored task record.
• Duplicate storage decision: Explain how the manager checks a proposed memory against existing memories and decides whether to skip storage.
  Why it matters: Checking before storage prevents the task-memory collection from accumulating duplicate records.
• Memory saving pipeline: Trace how save() moves from execution history to a stored ChromaDB memory, including its duplicate gate and storage representations.
  Why it matters: This sequence is the write path that turns a completed run into a reusable long-term memory.
• Typed memory search: Explain what search() returns and which parts of a ChromaDB query it delegates to ChromaDB.
  Why it matters: The read path gives callers relevant memories as structured records rather than raw database results.
• Agent memory access: Describe how the optional memory manager is made available to tools during an agent run.
  Why it matters: This connection lets tools use long-term memory without requiring it for every agent.

Key terms: _format_execution_history(), _extract_memory(), response_format=TaskMemory, LlmClient, _is_duplicate, DuplicateCheckResult, result.decision, SKIP, save(), ChromaDB

6.4.5 Retrieving memories

This section shows how an agent can retrieve stored memories on demand or inject them automatically into LLM requests.

• Optional retrieval of past experiences: Describe how a registered MemoryTool lets the agent search past problem-solving records when it decides they are relevant, and how execute() presents successful and failed approaches to the LLM.
  Why it matters: This is the alternative to retrieving memories on every LLM call.
• Automatic memory context injection: Explain how a tool can retrieve memories before every LLM call by using process_llm_request() to reuse execute() and append its results to the request, rather than splitting memory logic into a separate callback.
  Why it matters: The same memory logic can support automatic context without duplicating retrieval code.
• Tool visibility and request preparation: Explain how tool_definition controls whether the LLM can call MemoryTool, and describe the two changes to _prepare_llm_request() that allow injection-only tools to run without appearing in the LLM’s tool list.
  Why it matters: The agent can run memory injection without offering that tool as an action the LLM may choose.

Key terms: MemoryTool, execute(), tool-based retrieval, process_llm_request(), automatic injection, BaseTool, tool_definition, _prepare_llm_request()

6.5 Summary

This chapter summary connects three uses of agent memory with the context, session, and long-term-memory mechanisms that support them.