Chapter 4: The Session Service: Teaching your AI to remember
This chapter introduces the Session Service as the platform component that preserves conversational context, stores it as structured session data, and prepares it within model context-window limits.
After this chapter you will be able to
- Conversation memory purpose. Explain how the Session Service gives an otherwise stateless AI system conversation memory so it can interpret follow-up questions using earlier exchanges.
- Structured session relationships. Describe the relational structure of session data: users have conversations, conversations contain messages, and messages have roles and content; sessions are looked up by ID and messages are appended in sequence.
- Context-window management need. Explain why the Session Service must manage context windows: long histories can exceed a model's token limit, so the entire conversation cannot always be included in the next model request.
- Conversation message representation. Describe the platform's Message representation using the OpenAI-style role-and-content format, including the meanings of user, assistant, system, and tool roles. Explain that ordinary messages carry text content while assistant tool requests can carry tool_calls instead of text.
- Tool call result correlation. Trace the messages created when an assistant calls an external tool: the assistant records a tool call with a unique identifier, the tool result is stored as a tool-role message, and tool_call_id links that result back to its request. Explain why retaining both the request and result is necessary, especially when multiple tools run in parallel.
- Session identity and activity metadata. Describe a Session as a collection of messages plus session_id, user_id, created_at, and updated_at metadata. Explain how session ID identifies a conversation, user ID supports finding a user's sessions, and activity timestamps support identifying active sessions and cleanup.
- Five service operations. Describe the five Session Service operations and the two groups they form: session management uses GetOrCreateSession, ListSessions, and DeleteSession, while message operations use AddMessages and GetMessages.
- Session acquisition and selection. Explain how GetOrCreateSession accepts a user ID and optional session ID to return an existing session or create a fresh one, and how ListSessions lets a client display a user's sessions for the user to choose from. State that the client, not the Session Service, decides which session is active.
- Batched message history retrieval. Describe how AddMessages stores a repeated list of messages atomically, including user messages, tool calls, tool results, and assistant responses. Explain how GetMessages returns full or bounded history using optional limit and offset, with total_count indicating the complete number of messages.
- Pluggable session storage abstraction. Explain why the Session Service delegates persistence to a storage interface instead of choosing one required database. Describe how this lets teams substitute a backend without changing Session Service logic or client code.
- Session persistence request flow. Trace a session get-or-create request from a workflow through the gateway and Session Service to the configured storage backend. Describe the division of responsibility: the Session Service validates requests, enforces business rules, and formats responses, while the storage implementation handles persistence details.
- Session storage backend tradeoffs. Describe the main tradeoffs among relational SQL databases, in-memory stores, document databases, and managed cloud services for session data. Explain the characteristic benefit and limitation of each option discussed in the section.
- Requirements-driven backend selection. Explain why a team should select its initial session-storage backend from actual requirements, including compliance needs, query patterns, team expertise, and latency constraints. Describe why changing databases after launch remains difficult even with a storage abstraction.
- Session storage contract. Describe the SessionStorage abstract base class as the backend contract for session persistence. Explain that it covers creating and retrieving sessions, adding messages, and fetching message history.
- Backend-independent service operations. Explain that each storage-interface method maps directly to an earlier gRPC operation, with the same parameters and returned data. Describe how PostgreSQL, Redis, and DynamoDB can implement those calls differently while the Session Service simply calls the common methods and uses their results.
- PostgreSQL storage implementation. Describe how a concrete PostgreSQL class implements the previously defined SessionStorage interface for the Session Service. Explain that it implements all five interface methods while translating their operations into database operations.
- Database-independent service boundary. Explain why the Session Service receives a storage-class instance and invokes its methods without knowing PostgreSQL-specific details. Describe how this keeps the service dependent on the abstraction rather than on a particular storage technology.
- Session and message table roles. Describe how the sessions table holds conversation metadata and acts as the anchor for a conversation, while the messages table holds the individual conversation records. Explain that each message belongs to a session through a relationship back to its session.
- Message row design. Describe the key messages-table choices: an auto-incrementing SERIAL id for stable message ordering, TEXT for variable-length content, and JSONB for nested tool-call data. Explain that JSONB retains tool call IDs, function names, and arguments while remaining queryable and indexable.
- PostgreSQL storage adapter. Describe how the Python storage class implements the SessionStorage interface by translating service operations into PostgreSQL queries through psycopg2 cursors and returning Python objects.
- Session lookup or creation. Trace get_or_create_session: it returns an existing session when a supplied session ID is found, otherwise creates a session with a generated UUID, while ON CONFLICT makes concurrent creation with the same ID safe.
- Atomic message persistence. Explain how add_messages inserts a batch of session messages with executemany and one commit so the messages are atomic, then updates the session's updated_at timestamp.
- gRPC service translation role. Explain how the Session Service gRPC implementation translates incoming Protocol Buffer requests into storage-layer calls and translates their results into Protocol Buffer responses. State that this layer is thin wiring rather than a place for session-storage business logic.
- Storage dependency injection. Describe how the SessionServiceServicer receives an implementation of the SessionStorage interface through its constructor rather than directly creating PostgresSessionStorage. Explain that it can call methods such as get_or_create_session() and add_messages() without knowing whether the backend uses PostgreSQL, Redis, or another implementation.
- SessionClient SDK integration. Explain how SessionClient serves as the SDK-side counterpart to SessionServiceServicer, letting a developer call platform.sessions instead of interacting with gRPC directly. Describe how inheriting from BaseClient reuses gateway-channel setup and routing metadata configuration.
- Protocol Buffer stub behavior. Describe how a proto service definition becomes usable client code: the Protocol Buffer compiler generates message classes in sessions_pb2.py and the SessionServiceStub in sessions_pb2_grpc.py. Explain that the stub serializes a request, sends it over gRPC, waits for and deserializes the response, making an RPC resemble a local method call.
- Get-or-create request translation. Trace how get_or_create turns platform.sessions.get_or_create(user_id, optional session_id) into a GetOrCreateSessionRequest, invokes the stub with routing metadata, and converts the response into a Python Session object. Include that Unix-millisecond timestamps are converted to Python datetime objects and that metadata lets the gateway select the Session Service.
- Model-managed memory concept. Explain model-managed memory as an approach in which the LLM actively decides which important facts to save to external storage and retrieve later, rather than relying only on messages in its context window.
- Persistent structured facts. Describe how an important fact such as a patient's penicillin allergy can be explicitly saved through a save_memory tool as structured data separate from the conversation, making it available on future turns regardless of conversation length.
- Memory ownership and scope. Describe the memory association model: every memory has a user ID and may have an optional session ID. Explain how this supports session-scoped memory, user-level persistent memory, or both within one application.
- Memory entry structure. Describe a MemoryEntry as one stored fact with a key, a flexible value, a session_id for origin or optional session scoping, and timestamps that record its creation and updates. Explain that keys identify the kind of fact while values can hold simple or structured data.
- Mutable memory updates. Distinguish append-only messages from memories that can be updated in place. Explain that overwriting an existing memory prevents contradictory versions of a fact, while updated_at records when the change occurred.
- Concurrent memory update strategies. Explain why user-scoped memories can receive simultaneous updates from different sessions and compare last-write-wins, optimistic concurrency control using an updated_at condition, and append-for-later-reconciliation. State when the latter approaches are appropriate: when an overwrite would lose important information or when facts such as medications need merging.
- Memory storage operations. Describe the four memory operations added to SessionStorage: save or update a memory, retrieve one or many memories, delete a specific memory, and clear every memory for a user.
- User and session memory scope. Explain how user_id identifies memories and how optional session_id can either scope memories to a session or record their provenance for user-level persistence.
- Backend-independent memory persistence. Explain that applications use the same SessionStorage memory methods regardless of backend, while PostgreSQL and Redis can implement memory persistence differently based on their capabilities and access patterns.
- SessionClient memory operations. Describe how workflows use SessionClient memory methods to load memories for prompt context, save important facts, and delete or clear memories that are stale or incorrect. Explain that SessionClient wraps calls to the Session Service while handling transport details for the workflow.
- JSON memory value transport. Explain why memory values are JSON-encoded before a save_memory gRPC call and decoded after get_memory. Describe the boundary: gRPC receives strings, while the application receives native Python values such as strings, numbers, lists, and nested dictionaries.
- Model-managed memory workflow. Describe the three phases of the workflow: load existing memories, generate a response with memory tools available, and persist the conversation turn. Explain that the model decides which facts are worth remembering while the platform stores them.
- Cross-session patient memory. Explain how memories are loaded by user and included in the system prompt, so facts saved in one session remain available in later sessions. Use the allergy and medication example to show how retained facts can affect a later response.
- Memories versus conversation history. Distinguish memories from conversation messages: memories preserve structured critical facts—the "what"—whereas messages retain the discussion context—the "how." Explain why the two approaches complement each other when history must be truncated or summarized for a limited context window.
- Context window selection problem. Explain why faithfully stored session history cannot usually all be included in a model request, and describe the competing information that consumes a context window.
- Flexible platform context role. Describe the division of responsibility for context management: the platform provides reusable building blocks and strategies, while each workflow chooses and combines them according to its application needs.
- Context engineering purpose. Define context engineering as selecting exactly the right information for each model call, and explain why it is especially important for long-running applications.
- Shared context window budget. Explain that a model's context window is a fixed limit shared by conversation history and other prompt components, rather than space reserved entirely for the conversation.
- Conversation history selection. Describe why the Session Service must make intelligent choices about which conversation messages to include when a conversation outgrows its allocated history budget.
- Simple truncation procedure. Describe how simple truncation keeps the newest messages that fit within the token budget by working backward from the latest message, accumulating tokens until the limit is reached, and dropping all older messages.
- Truncation tradeoffs and limits. Explain why simple truncation is fast, predictable, and inexpensive, and distinguish the short, self-contained interactions where it works well from context-dependent conversations where dropping older messages can remove critical facts.
- History compression tradeoff. Explain how summarization replaces older verbatim messages with a much shorter summary that preserves essential facts, and contrast this with simply truncating old history. State that summarization extends effective context but requires an LLM call, adding cost and latency.
- Domain-directed summary prompts. Describe why a generic summarization prompt can omit critical details and how an application-specific prompt directs the model to preserve the facts that matter. Give the section's examples: medical conditions, allergies, and safety concerns for healthcare, or commitments, deadlines, and agreed terms for legal work.
- Summary caching and scheduling. Describe the implementation pattern for usable conversation summaries: mark compressed context clearly, cache it with a hash of the messages it covers, and regenerate it only when those underlying messages change. Explain the freshness tradeoff among summarizing on every retrieval, periodically after a threshold, and asynchronously in the background.
- Three-tier memory hierarchy. Describe the three tiers of hierarchical memory: model-managed structured memories for critical facts, a rolling summary for older conversation context, and verbatim recent messages for the current conversational flow. Explain why each tier receives a different level of fidelity and compression.
- Context ordering for attention. Explain the lost-in-the-middle phenomenon and how it affects context assembly: place critical memories and safety information at the beginning, recent messages at the end, and less essential material in the middle.
- Cross-session conversational memory. Explain how retrieval-augmented memory lets an assistant answer questions whose relevant information exists in a user’s previous sessions rather than the current conversation.
- Conversation retrieval pipeline. Describe the retrieval-augmented memory flow: index session messages as embeddings, turn the user’s current query into a search query, retrieve and rank relevant past results, and include the top matches in the current context within a token budget.
- Context strategy selection. Explain how to select truncation, summarization, model-managed memory, or retrieval-augmented memory according to how users converse and what information the application must retain.
- Service-based context assembly. Describe how workflows select and compose context strategies while the Session Service assembles session history, and explain the Data Service's role in cross-session semantic retrieval.
- Context engineering purpose. Explain context engineering as assembling exactly the information needed for each model call while accounting for token budgets and attention patterns.
What it builds on
- Thinking from first principles: what AI applications actually need (chapter 1) · The need for context-aware intelligence as a fundamental AI application capability.
- The conversation memory challenge (chapter 2) · The developer-facing conversation memory challenge.
- Generating responses (chapter 3) · Model-response generation requests that require appropriately prepared input.
- OpenAI message format as platform standard (chapter 3) · The choice of OpenAI message format as the platform standard.
- The internal communication challenge (chapter 2) · The platform's internal service-to-service communication design.
- Providing access to services (chapter 2) · The SDK's role in providing access to platform services.
- The SDK foundation (chapter 2) · The SDK foundation that hides service communication behind simple client calls.
- The communication protocol (chapter 2) · The platform communication protocol.
Section by section
Introduction
This chapter introduces the Session Service as the platform component that preserves conversational context, stores it as structured session data, and prepares it within model context-window limits.
- Conversation memory purpose: Explain how the Session Service gives an otherwise stateless AI system conversation memory so it can interpret follow-up questions using earlier exchanges.
Why it matters: Conversation memory is what lets an assistant maintain a coherent dialogue instead of treating every request as unrelated. - Structured session relationships: Describe the relational structure of session data: users have conversations, conversations contain messages, and messages have roles and content; sessions are looked up by ID and messages are appended in sequence.
Why it matters: A clear data structure makes conversation history reliable to store, retrieve, and extend like other backend user data. - Context-window management need: Explain why the Session Service must manage context windows: long histories can exceed a model's token limit, so the entire conversation cannot always be included in the next model request.
Why it matters: Without context-window management, a growing conversation eventually becomes too large for the model to process.
Key terms: Session Service, conversation memory, stateless AI system, context-aware intelligence, sessions, conversations, messages, roles, content, session ID
4.1 What a session contains
Defines the lightweight session and message data model that preserves conversational context, including tool-use exchanges, for efficient AI memory.
- Conversation message representation: Describe the platform's Message representation using the OpenAI-style role-and-content format, including the meanings of user, assistant, system, and tool roles. Explain that ordinary messages carry text content while assistant tool requests can carry tool_calls instead of text.
Why it matters: A consistent message format lets the platform preserve and replay the context a model needs to continue a conversation coherently. - Tool call result correlation: Trace the messages created when an assistant calls an external tool: the assistant records a tool call with a unique identifier, the tool result is stored as a tool-role message, and tool_call_id links that result back to its request. Explain why retaining both the request and result is necessary, especially when multiple tools run in parallel.
Why it matters: Without the complete linked tool exchange, later model turns cannot reliably understand what action was taken or what information it returned. - Session identity and activity metadata: Describe a Session as a collection of messages plus session_id, user_id, created_at, and updated_at metadata. Explain how session ID identifies a conversation, user ID supports finding a user's sessions, and activity timestamps support identifying active sessions and cleanup.
Why it matters: This metadata makes conversations addressable, attributable to users, and manageable over time without complicating the conversation history. - Lightweight hot-path session design: Justify the deliberately simple, flat list of messages and lightweight session metadata: every turn fetches history and appends new messages, so sessions must be fast to read and write. State that message timestamps can later help decide which context is most relevant when conversations become long.
Why it matters: Session storage sits on every conversational turn, so a simple model protects responsiveness while leaving room for later context management.
Key terms: Message, role, content, user, assistant, system, tool_calls, tool_call_id, tool, Function
4.2 The session service contract
Defines the gRPC and Protocol Buffers contract through which workflows create or resume sessions, manage their lifecycle, and record or retrieve conversation history.
- Five service operations: Describe the five Session Service operations and the two groups they form: session management uses GetOrCreateSession, ListSessions, and DeleteSession, while message operations use AddMessages and GetMessages.
Why it matters: These operations give every workflow one consistent way to persist and recover conversational state. - Session acquisition and selection: Explain how GetOrCreateSession accepts a user ID and optional session ID to return an existing session or create a fresh one, and how ListSessions lets a client display a user's sessions for the user to choose from. State that the client, not the Session Service, decides which session is active.
Why it matters: A combined get-or-create call keeps workflows simple and avoids races caused by separate lookup and creation calls. - Batched message history retrieval: Describe how AddMessages stores a repeated list of messages atomically, including user messages, tool calls, tool results, and assistant responses. Explain how GetMessages returns full or bounded history using optional limit and offset, with total_count indicating the complete number of messages.
Why it matters: Workflows need complete ordered conversational context while still being able to request only the recent history that fits a model context window. - Protocol Buffer contract evolution: Explain that the Session Service uses gRPC and Protocol Buffers, whose generated code serializes requests and responses while the SDK converts them to Python objects. Describe how adding a new proto field with a new field number lets the service evolve without breaking existing clients because missing fields are handled gracefully.
Why it matters: A stable wire contract lets the service gain capabilities independently of workflows that have not yet upgraded.
Key terms: GetOrCreateSession, ListSessions, DeleteSession, AddMessages, GetMessages, session management, GetOrCreateSessionRequest, GetOrCreateSessionResponse, Session, user ID
4.3 Storage abstraction
This section explains why the Session Service separates session logic from persistence through a pluggable storage abstraction so teams can choose databases that fit their requirements.
- Pluggable session storage abstraction: Explain why the Session Service delegates persistence to a storage interface instead of choosing one required database. Describe how this lets teams substitute a backend without changing Session Service logic or client code.
Why it matters: A storage abstraction prevents one team's durability, cost, scalability, or performance needs from becoming every team's constraint. - Storage requirement tradeoffs: Describe why different session workloads require different storage backends, including tradeoffs among auditability and ACID compliance, speed and durability, automatic expiration, serverless scaling, and cost. State that no single database is optimal for every requirement.
Why it matters: Choosing storage based on workload requirements avoids forcing inappropriate durability, operational, performance, or cost tradeoffs on an application. - Session persistence request flow: Trace a session get-or-create request from a workflow through the gateway and Session Service to the configured storage backend. Describe the division of responsibility: the Session Service validates requests, enforces business rules, and formats responses, while the storage implementation handles persistence details.
Why it matters: This boundary keeps the service behavior consistent while allowing persistence technology to change independently.
Key terms: storage abstraction, storage interface, pluggable storage layer, persistence, Session Service, ACID compliance, audit trails, durability, TTL, PostgreSQL
4.3.1 Choosing the right database
This section compares session-data storage backends, explains how to choose among their tradeoffs, and establishes PostgreSQL as the book’s implementation example.
- Session storage backend tradeoffs: Describe the main tradeoffs among relational SQL databases, in-memory stores, document databases, and managed cloud services for session data. Explain the characteristic benefit and limitation of each option discussed in the section.
Why it matters: Choosing session storage means balancing reliability, speed, flexibility, operational effort, and provider dependence. - Requirements-driven backend selection: Explain why a team should select its initial session-storage backend from actual requirements, including compliance needs, query patterns, team expertise, and latency constraints. Describe why changing databases after launch remains difficult even with a storage abstraction.
Why it matters: An early storage choice can become costly to reverse once production data and operational dependencies accumulate. - PostgreSQL implementation choice: State why the book implements PostgreSQL in detail: it is a common production choice when reliability is needed. Explain that the resulting abstraction and patterns can be extended to other SQL databases and later alternatives.
Why it matters: PostgreSQL is the concrete example, but the design is intended to avoid permanently coupling the platform to it.
Key terms: relational (SQL) databases, PostgreSQL, in-memory stores, Redis, document databases, MongoDB, storage abstraction, compliance needs, query patterns, team expertise
4.3.2 The storage interface
This section defines a minimal storage abstraction that lets the Session Service use interchangeable persistence backends without changing its service or application logic.
- Session storage contract: Describe the SessionStorage abstract base class as the backend contract for session persistence. Explain that it covers creating and retrieving sessions, adding messages, and fetching message history.
Why it matters: A clear storage contract separates the Session Service's behavior from the database technology that persists its data. - Backend-independent service operations: Explain that each storage-interface method maps directly to an earlier gRPC operation, with the same parameters and returned data. Describe how PostgreSQL, Redis, and DynamoDB can implement those calls differently while the Session Service simply calls the common methods and uses their results.
Why it matters: Backend independence lets the service retain one behavior while each storage technology translates requests into its own queries or API calls. - Abstraction-driven backend flexibility: Explain how the storage abstraction supports database-free testing, gradual backend migration, and future support for new databases. State that an in-memory Python-dictionary implementation can test service logic, while a production backend can be replaced by implementing the same class without modifying application code.
Why it matters: The abstraction reduces the cost and risk of testing, scaling changes, and adopting future storage technologies.
Key terms: SessionStorage, abstract base class, session persistence, message history, gRPC operations, PostgreSQL, Redis, DynamoDB, SQL queries, in-memory implementation
4.4 Implementing the Session Service backend
This section introduces the PostgreSQL-backed Session Service implementation and the storage-abstraction patterns it demonstrates.
- PostgreSQL storage implementation: Describe how a concrete PostgreSQL class implements the previously defined SessionStorage interface for the Session Service. Explain that it implements all five interface methods while translating their operations into database operations.
Why it matters: This separation lets the service use session storage without being coupled to a particular database. - Storage implementation responsibilities: Explain the general responsibilities demonstrated by a storage implementation: translating interface methods into database operations, handling errors, and managing database connections. Distinguish these reusable structural responsibilities from PostgreSQL-specific SQL queries.
Why it matters: These responsibilities are the common implementation pattern that allows storage backends to be changed without changing the service contract. - Database-independent service boundary: Explain why the Session Service receives a storage-class instance and invokes its methods without knowing PostgreSQL-specific details. Describe how this keeps the service dependent on the abstraction rather than on a particular storage technology.
Why it matters: A database-independent service boundary makes the platform easier to maintain and lets implementations vary behind a stable contract.
Key terms: PostgreSQL, SessionStorage, storage interface, Session Service, database operations, error handling, connection management, SQL queries, storage implementation
4.4.1 The Database Schema
This section designs a PostgreSQL schema that stores conversation metadata and message history efficiently while preserving their relationship.
- Session and message table roles: Describe how the sessions table holds conversation metadata and acts as the anchor for a conversation, while the messages table holds the individual conversation records. Explain that each message belongs to a session through a relationship back to its session.
Why it matters: Separating session metadata from message content makes conversations resumable and lets message histories grow independently. - Lookup indexes and identifiers: Explain why session ID is a unique primary key with an automatic index and why user ID receives a separate non-unique index. Describe how these indexes support fast session lookup by ID and fast retrieval of all sessions for one user.
Why it matters: Indexes prevent common session lookups from degrading into scans of every conversation as the database grows. - Message row design: Describe the key messages-table choices: an auto-incrementing SERIAL id for stable message ordering, TEXT for variable-length content, and JSONB for nested tool-call data. Explain that JSONB retains tool call IDs, function names, and arguments while remaining queryable and indexable.
Why it matters: These column choices let the platform retain complete model messages, order them reliably, and support later queries about tool use. - Cascading message deletion: Explain that the messages table has a foreign key to its session and that ON DELETE CASCADE automatically removes a session's messages when the session is deleted. Contrast this with manually deleting messages first or leaving orphaned records.
Why it matters: Cascading deletion keeps session cleanup consistent and prevents message rows from being left without a parent conversation.
Key terms: sessions table, messages table, session ID, user ID, foreign key, primary key, index, CREATE INDEX, CREATE INDEX CONCURRENTLY, SERIAL
4.4.2 The Storage Implementation
This section implements PostgreSQL-backed session storage, covering session creation, atomic message writes, paginated retrieval, lifecycle operations, and row-to-object conversion.
- PostgreSQL storage adapter: Describe how the Python storage class implements the SessionStorage interface by translating service operations into PostgreSQL queries through psycopg2 cursors and returning Python objects.
Why it matters: This adapter isolates database-specific work so the rest of the Session Service can use a consistent storage interface. - Session lookup or creation: Trace get_or_create_session: it returns an existing session when a supplied session ID is found, otherwise creates a session with a generated UUID, while ON CONFLICT makes concurrent creation with the same ID safe.
Why it matters: Workflows can handle both returning users and new conversations with one session operation without duplicate-key failures during concurrent requests. - Atomic message persistence: Explain how add_messages inserts a batch of session messages with executemany and one commit so the messages are atomic, then updates the session's updated_at timestamp.
Why it matters: A conversation turn can include a user message, tool activity, and a final response, all of which must be recorded together without leaving partial history. - Paginated ordered message retrieval: Describe how get_messages uses optional limit and offset to return a chunk of a session's messages together with the total message count, ordered by auto-incrementing ID to preserve conversation sequence.
Why it matters: Long conversations can be displayed or processed in manageable chunks while still preserving their complete order and indicating whether more history exists. - Session lifecycle and object conversion: State that delete_session deletes the session and, through ON DELETE CASCADE, its messages; list_sessions returns a user's sessions ordered by updated_at descending; and helper methods convert database rows at the storage boundary into Session and related Python objects.
Why it matters: These operations provide clean session lifecycle management and keep database-specific representations from leaking through the application.
Key terms: SessionStorage, psycopg2, connection string, cursor, SQL queries, storage layer, get_or_create_session, session ID, user ID, UUID
4.4.3 The gRPC service implementation
This section implements the Session Service gRPC layer as a thin, dependency-injected bridge between API Gateway Protocol Buffer requests and storage operations.
- gRPC service translation role: Explain how the Session Service gRPC implementation translates incoming Protocol Buffer requests into storage-layer calls and translates their results into Protocol Buffer responses. State that this layer is thin wiring rather than a place for session-storage business logic.
Why it matters: Keeping network translation separate from persistence logic makes the service easier to understand, test, and change. - Gateway-to-service request path: Trace the path of a workflow call to platform.sessions.get_or_create() from the SDK through the API Gateway to a healthy Session Service container. Describe how the gateway uses x-target-service metadata to route the request to the sessions service.
Why it matters: Understanding the hops clarifies how a workflow reaches the correct internal platform service. - Storage dependency injection: Describe how the SessionServiceServicer receives an implementation of the SessionStorage interface through its constructor rather than directly creating PostgresSessionStorage. Explain that it can call methods such as get_or_create_session() and add_messages() without knowing whether the backend uses PostgreSQL, Redis, or another implementation.
Why it matters: Dependency injection lets the network service remain independent of a particular database backend, so storage can be replaced without changing service code.
Key terms: gRPC service, Protocol Buffer requests, Protocol Buffer responses, SessionServiceServicer, SessionStorage, platform.sessions.get_or_create(), API Gateway, x-target-service, Session Service container, Protocol Buffer request
4.5 Integrating with the SDK
This section integrates the Session Service into the SDK so developers can use simple Python calls that are translated into routed gRPC requests and Python session objects.
- SessionClient SDK integration: Explain how SessionClient serves as the SDK-side counterpart to SessionServiceServicer, letting a developer call platform.sessions instead of interacting with gRPC directly. Describe how inheriting from BaseClient reuses gateway-channel setup and routing metadata configuration.
Why it matters: This client layer gives developers a simple Python interface while preserving the platform's shared service communication architecture. - Protocol Buffer stub behavior: Describe how a proto service definition becomes usable client code: the Protocol Buffer compiler generates message classes in sessions_pb2.py and the SessionServiceStub in sessions_pb2_grpc.py. Explain that the stub serializes a request, sends it over gRPC, waits for and deserializes the response, making an RPC resemble a local method call.
Why it matters: Understanding the generated stub explains how an ordinary-looking SDK method call safely crosses the network to the Session Service. - Get-or-create request translation: Trace how get_or_create turns platform.sessions.get_or_create(user_id, optional session_id) into a GetOrCreateSessionRequest, invokes the stub with routing metadata, and converts the response into a Python Session object. Include that Unix-millisecond timestamps are converted to Python datetime objects and that metadata lets the gateway select the Session Service.
Why it matters: This translation pattern is what lets workflow code create or resume sessions while the SDK handles wire formats, routing, and Python-friendly results. - Lazy session client creation: Trace the first session call in a workflow: GenAIPlatform starts with sessions set to None, accessing platform.sessions lazily creates SessionClient and opens its gRPC channel, then the request travels through the gateway, SessionServiceServicer, storage layer, and PostgreSQL before returning as a Session object.
Why it matters: Lazy creation avoids making a session-service network connection until a workflow actually needs session capabilities.
Key terms: SessionClient, SessionServiceServicer, BaseClient, GenAIPlatform, API gateway, routing metadata, proto file, Protocol Buffer compiler, sessions_pb2.py, sessions_pb2_grpc.py
4.6 Model-managed memory
This section introduces model-managed memory, where an assistant explicitly saves important facts outside the conversation history so they persist across long interactions.
- Model-managed memory concept: Explain model-managed memory as an approach in which the LLM actively decides which important facts to save to external storage and retrieve later, rather than relying only on messages in its context window.
Why it matters: It lets an assistant retain important information even when a conversation becomes too long to fit in the model's context. - Persistent structured facts: Describe how an important fact such as a patient's penicillin allergy can be explicitly saved through a save_memory tool as structured data separate from the conversation, making it available on future turns regardless of conversation length.
Why it matters: Persistent structured facts prevent critical details from being lost through history truncation or summarization. - Session memory store extension: State that model-managed memory extends the existing Session Service with a memory store, because the service already manages sessions and messages and can naturally also manage persistent memories.
Why it matters: Adding memory to the Session Service keeps related forms of conversation state behind one platform service.
Key terms: model-managed memory, LLM, context window, external storage, RAM, disk, save_memory tool, memory store, structured data, conversation history
4.6.1 Where memory lives
Defines how memories are associated with users and optionally sessions so applications can support temporary context, persistent facts, provenance, and application-specific deletion policies.
- Memory ownership and scope: Describe the memory association model: every memory has a user ID and may have an optional session ID. Explain how this supports session-scoped memory, user-level persistent memory, or both within one application.
Why it matters: Memory scope determines whether an assistant treats a fact as temporary conversational context or as something worth remembering about a user over time. - Session provenance and deletion policy: Explain that a session ID can record the conversation that surfaced a persistent memory as provenance for debugging and auditability. Describe why deletion of session-associated memories is an application policy rather than an automatic platform behavior.
Why it matters: Keeping provenance while letting applications choose cleanup behavior makes memory usable in domains with very different retention requirements.
Key terms: user ID, session ID, session-scoped memory, user-level persistence, memory model, provenance, debugging, auditability, cascade, session lifecycle
4.6.2 The memory model
This section defines the structure and update behavior of model-managed memory entries, including strategies for concurrent changes to user-scoped facts.
- Memory entry structure: Describe a MemoryEntry as one stored fact with a key, a flexible value, a session_id for origin or optional session scoping, and timestamps that record its creation and updates. Explain that keys identify the kind of fact while values can hold simple or structured data.
Why it matters: A clear memory record lets applications retain useful user facts without restricting those facts to one data shape. - Mutable memory updates: Distinguish append-only messages from memories that can be updated in place. Explain that overwriting an existing memory prevents contradictory versions of a fact, while updated_at records when the change occurred.
Why it matters: Treating durable user facts as mutable records keeps an assistant's stored preferences current rather than leaving it with conflicting history. - Concurrent memory update strategies: Explain why user-scoped memories can receive simultaneous updates from different sessions and compare last-write-wins, optimistic concurrency control using an updated_at condition, and append-for-later-reconciliation. State when the latter approaches are appropriate: when an overwrite would lose important information or when facts such as medications need merging.
Why it matters: Choosing a conflict strategy prevents simultaneous agents or sessions from silently destroying important user information.
Key terms: MemoryEntry, key, value, session_id, updated_at, append-only log, memories, overwrite, last-write-wins, optimistic concurrency control
4.6.3 Extending the storage interface for memories
This section extends the SessionStorage abstraction with consistent memory persistence operations, user/session scoping, privacy deletion, and backend-independent implementations.
- Memory storage operations: Describe the four memory operations added to SessionStorage: save or update a memory, retrieve one or many memories, delete a specific memory, and clear every memory for a user.
Why it matters: These operations give applications a standard way to persist, retrieve, remove, and fully erase model-managed memories. - User and session memory scope: Explain how user_id identifies memories and how optional session_id can either scope memories to a session or record their provenance for user-level persistence.
Why it matters: Correct use of user and session identifiers determines whether an assistant remembers something only within one conversation or across a user's conversations. - Backend-independent memory persistence: Explain that applications use the same SessionStorage memory methods regardless of backend, while PostgreSQL and Redis can implement memory persistence differently based on their capabilities and access patterns.
Why it matters: A stable storage interface lets teams choose an existing database now and migrate storage later without rewriting application memory calls.
Key terms: SessionStorage, save_memory, get_memory, delete_memory, clear_user_memory, user_id, session_id, session-scoped memory, user-level persistence, provenance
4.6.4 SDK Integration
This section integrates model-managed memory into the SDK by exposing SessionClient methods that serialize flexible memory values for gRPC transport and return native Python objects to workflows.
- SessionClient memory operations: Describe how workflows use SessionClient memory methods to load memories for prompt context, save important facts, and delete or clear memories that are stale or incorrect. Explain that SessionClient wraps calls to the Session Service while handling transport details for the workflow.
Why it matters: These SDK methods let application code use long-term memory without directly managing Session Service gRPC calls. - JSON memory value transport: Explain why memory values are JSON-encoded before a save_memory gRPC call and decoded after get_memory. Describe the boundary: gRPC receives strings, while the application receives native Python values such as strings, numbers, lists, and nested dictionaries.
Why it matters: JSON serialization preserves flexible memory values across the service boundary while keeping workflow code convenient to use. - Memory write audit trail: Justify logging every save_memory call together with the context that triggered it, treating memory writes as audit events. Explain that this trail supports answering why the system appeared to know a particular fact.
Why it matters: An audit trail makes persistent AI memory explainable when users or operators question where a remembered fact came from.
Key terms: SessionClient, Session Service, save_memory, get_memory, delete_memory, clear_user_memory, JSON encoding, JSON decoding, MemoryEntry, gRPC
4.6.5 A complete workflow example
This section demonstrates a healthcare-assistant workflow that loads persistent memories, lets the model save important new facts, and keeps conversation history separate from durable knowledge.
- Model-managed memory workflow: Describe the three phases of the workflow: load existing memories, generate a response with memory tools available, and persist the conversation turn. Explain that the model decides which facts are worth remembering while the platform stores them.
Why it matters: This pattern gives an AI application durable, model-selected knowledge without making workflow code decide every fact to save. - Cross-session patient memory: Explain how memories are loaded by user and included in the system prompt, so facts saved in one session remain available in later sessions. Use the allergy and medication example to show how retained facts can affect a later response.
Why it matters: Persistent facts let an assistant retain safety-critical and personalized context even when a user returns months later in a new session. - Memories versus conversation history: Distinguish memories from conversation messages: memories preserve structured critical facts—the "what"—whereas messages retain the discussion context—the "how." Explain why the two approaches complement each other when history must be truncated or summarized for a limited context window.
Why it matters: Keeping durable facts separate from full chat history preserves important information without requiring the model to carry every past message. - Persistent-memory fit: Justify when model-managed memory is useful: applications with facts that must persist reliably, safety-critical context, or personalization benefit, while a one-off Q&A bot usually gains little. Give examples such as allergies, medications, preferences, or key decisions.
Why it matters: Choosing persistent memory only where durable context has value avoids adding unnecessary complexity to simple interactions.
Key terms: model-managed memory, save_memory, Session Service, conversation turn, get_memory, system prompt, memories, session, messages, conversation history
4.7 Managing context windows
This introduction frames context engineering as the problem of selecting the right session information for a limited model context window, with the platform supplying flexible strategies and workflows choosing how to use them.
- Context window selection problem: Explain why faithfully stored session history cannot usually all be included in a model request, and describe the competing information that consumes a context window.
Why it matters: A useful AI response depends on fitting the most relevant information into the model's limited context. - Flexible platform context role: Describe the division of responsibility for context management: the platform provides reusable building blocks and strategies, while each workflow chooses and combines them according to its application needs.
Why it matters: This avoids making every team rebuild common context-management capabilities without forcing unrelated applications into one policy. - Context engineering purpose: Define context engineering as selecting exactly the right information for each model call, and explain why it is especially important for long-running applications.
Why it matters: Choosing relevant facts and conversation material helps the model act on what matters rather than merely on what was said most recently.
Key terms: context window, system prompts, tool definitions, retrieved documents, token budget, Session Service, platform, workflows, building blocks, reusable strategies
4.7.1 The token budget problem
This section frames the context window as a shared token budget and explains why the Session Service must choose which conversation messages to include.
- Shared context window budget: Explain that a model's context window is a fixed limit shared by conversation history and other prompt components, rather than space reserved entirely for the conversation.
Why it matters: A session history can only be managed well when you account for everything else that occupies the model's context window. - Conversation history selection: Describe why the Session Service must make intelligent choices about which conversation messages to include when a conversation outgrows its allocated history budget.
Why it matters: Without a selection strategy, long-running conversations eventually exceed the space available for history and cannot be sent intact to the model.
Key terms: context window, token budget, conversation history, system prompt, tool definitions, retrieval-augmented generation (RAG), Session Service, message selection
4.7.2 Simple truncation
This section explains simple truncation as a fast way to fit conversation history into a token budget, while showing why it can lose important older context.
- Simple truncation procedure: Describe how simple truncation keeps the newest messages that fit within the token budget by working backward from the latest message, accumulating tokens until the limit is reached, and dropping all older messages.
Why it matters: This is the simplest baseline for controlling how much conversation history is sent to a model. - Truncation tradeoffs and limits: Explain why simple truncation is fast, predictable, and inexpensive, and distinguish the short, self-contained interactions where it works well from context-dependent conversations where dropping older messages can remove critical facts.
Why it matters: A memory policy that blindly drops old history can cause an assistant to contradict prior statements or overlook safety-critical information.
Key terms: simple truncation, token budget, context window, LLM calls, safety-critical information
4.7.3 Compressing history with summarization
This subsection explains how LLM-generated summaries compress older conversation history to extend effective context while managing fidelity, cost, latency, and freshness.
- History compression tradeoff: Explain how summarization replaces older verbatim messages with a much shorter summary that preserves essential facts, and contrast this with simply truncating old history. State that summarization extends effective context but requires an LLM call, adding cost and latency.
Why it matters: Summarization lets an AI retain important earlier context without sending an entire long conversation to the model. - Domain-directed summary prompts: Describe why a generic summarization prompt can omit critical details and how an application-specific prompt directs the model to preserve the facts that matter. Give the section's examples: medical conditions, allergies, and safety concerns for healthcare, or commitments, deadlines, and agreed terms for legal work.
Why it matters: The summarization prompt determines which information survives compression, so vague prompts can remove facts the application needs. - Summary caching and scheduling: Describe the implementation pattern for usable conversation summaries: mark compressed context clearly, cache it with a hash of the messages it covers, and regenerate it only when those underlying messages change. Explain the freshness tradeoff among summarizing on every retrieval, periodically after a threshold, and asynchronously in the background.
Why it matters: Caching and selecting an update strategy prevent summary generation from adding unnecessary latency and cost to every request.
Key terms: summarization, effective context, verbatim messages, truncation, token count, summarization prompt, generic summaries, medical conditions, allergies, safety concerns
4.7.4 Hierarchical memory
This section explains how hierarchical memory combines durable facts, compressed history, and verbatim recent messages while allocating and ordering context to preserve what models need most.
- Three-tier memory hierarchy: Describe the three tiers of hierarchical memory: model-managed structured memories for critical facts, a rolling summary for older conversation context, and verbatim recent messages for the current conversational flow. Explain why each tier receives a different level of fidelity and compression.
Why it matters: A tiered design preserves essential information and current conversational nuance without spending the entire context window on old messages. - Context ordering for attention: Explain the lost-in-the-middle phenomenon and how it affects context assembly: place critical memories and safety information at the beginning, recent messages at the end, and less essential material in the middle.
Why it matters: Ordering context around model attention patterns makes critical facts and the active conversation more likely to influence the response. - Tier budget allocation: Describe how token budget allocation across memories, summaries, and recent messages is an application-specific tuning decision. State the suggested starting allocation of roughly 10% for memories, 20% for summaries, and 70% for recent messages, and explain how use cases can shift it.
Why it matters: Allocating tokens by what users and applications most need prevents a fixed memory policy from degrading important interactions.
Key terms: hierarchical memory, model-managed memories, rolling summary, verbatim recent messages, conversational fidelity, lost in the middle, context assembly, critical facts, safety information, token budget
4.7.5 Retrieval-augmented memory
Retrieval-augmented memory extends an assistant beyond the current session by searching a user’s past conversations and adding relevant results to the current context.
- Cross-session conversational memory: Explain how retrieval-augmented memory lets an assistant answer questions whose relevant information exists in a user’s previous sessions rather than the current conversation.
Why it matters: It enables an assistant to use the history of its relationship with a user, not just the messages in front of it. - Conversation retrieval pipeline: Describe the retrieval-augmented memory flow: index session messages as embeddings, turn the user’s current query into a search query, retrieve and rank relevant past results, and include the top matches in the current context within a token budget.
Why it matters: This flow provides relevant long-term context without placing an entire conversation history into every model request. - Conversational RAG pattern: Relate retrieval-augmented memory to retrieval-augmented generation (RAG), distinguishing its conversation-history search corpus from the document corpus used in standard RAG.
Why it matters: Seeing this as RAG for conversations lets the platform reuse familiar semantic-search, ranking, filtering, and context-selection techniques.
Key terms: retrieval-augmented memory, past sessions, current conversation, conversation history, Search Service, embeddings, semantic search, search query, ranked and filtered, token budget
4.7.6 Putting it together
This section shows how applications choose and combine context-management strategies based on user behavior, safety needs, token budgets, and cross-session recall requirements.
- Context strategy selection: Explain how to select truncation, summarization, model-managed memory, or retrieval-augmented memory according to how users converse and what information the application must retain.
Why it matters: Choosing only the needed strategy avoids unnecessary complexity while preserving context that affects answer quality or safety. - Incremental memory architecture: Describe the recommended incremental path for context management: start with truncation, add summarization when conversations become long, add memory for critical facts, and add retrieval only when cross-session context improves the experience.
Why it matters: An incremental approach lets teams match memory complexity to demonstrated user needs rather than building every capability prematurely. - Service-based context assembly: Describe how workflows select and compose context strategies while the Session Service assembles session history, and explain the Data Service's role in cross-session semantic retrieval.
Why it matters: This division lets workflows control context choices without having to implement history assembly and search internals themselves. - Context engineering purpose: Explain context engineering as assembling exactly the information needed for each model call while accounting for token budgets and attention patterns.
Why it matters: The quality of long-running AI applications depends on supplying relevant context, not only on writing a good prompt.
Key terms: simple truncation, summarization, model-managed memory, retrieval-augmented memory, token budget, GetMessagesRequest, strategy field, Session Service, Data Service, hierarchical strategy
4.8 Summary
This summary recaps how the Session Service preserves conversational state, supports durable memory, and manages limited model context windows.
Where this is used later
- Teaching Claw to remember conversations (chapter 10) · Uses the Session Service to make the Claw assistant remember conversations.
- From conversation to action: tools and the agent loop (chapter 10) · The assistant's agent loop needs tool calls and results as part of its conversational context.
- Choosing a vector store backend (chapter 5) · The existing PostgreSQL-backed Session Service infrastructure provides the operational context for reusing PostgreSQL.
- (chapter intro) (chapter 5) · The structured database schema used for session information.
- Service contract and complete retrieval flow (chapter 5) · Session Service integration with the SDK.
- Long-term memory: remembering across sessions (chapter 10) · Long-term memory applies this capability across assistant sessions.
- Domain-specific telemetry (chapter 7) · Domain-specific telemetry can record service events that explain AI behavior.
- Seeing what Claw does: observability (chapter 10) · Observability helps explain what the assistant did and why it produced behavior based on remembered information.
Concepts explained in this chapter
- memoryModel-managed memory is a capability in which an AI model selects, stores, and retrieves information from persistent memory outside its immediate context window.
- messageA message is a structured conversational item containing a role and content, optionally accompanied by metadata for tool interactions.
- session serviceA session service is a platform service that stores and manages conversational state, user preferences, interactions, and conversation history across sessions.
- sessionA session is a structured record representing one conversation between a user and an AI application, including its messages and identifying metadata.
- storage interfaceA storage interface is an abstract contract that defines operations for persisting and retrieving application data independently of a backend.
- OpenAI message formatThe OpenAI message format is a message representation in which a conversation is an array of objects containing fields such as role and content, with additional fields for tool invocation results.
- on delete cascadeON DELETE CASCADE is a relational database constraint action that automatically deletes dependent rows when their referenced row is deleted.
- context engineeringContext engineering is the deliberate assembly of the information supplied to a model call within constraints such as token capacity and attention behavior.
- retrieval-augmented memoryRetrieval-augmented memory retrieves relevant information from prior sessions and incorporates it into a current interaction.
- dialogueA conversation is a structured sequence of messages exchanged between a user, an assistant, and the system instructions governing the interaction.
- foreign keyA foreign key is a database field or set of fields that references a key in another table to enforce referential integrity.
- jsonbJSONB is PostgreSQL's binary JSON data type for storing structured JSON values with database indexing and querying support.
- truncationTruncation is a context-management technique that removes older or excess messages so the remaining content fits within a token budget.
- hierarchical memoryHierarchical memory is a memory architecture that organizes information into layers with different levels of structure, compression, recency, and retention priority.
- rolling summaryA rolling summary is a continuously updated compressed representation of older conversation content that preserves its essential narrative.
- serializationSerialization is the conversion of in-memory data structures into a transferable or storable representation.
- contractAn API is a defined interface through which software components communicate and invoke capabilities, including interfaces exposed by external systems.
- session managementSession management is the creation, persistence, retrieval, and expiration of state associated with user or application sessions.
- tool usetool_calls is a message field containing an assistant's requested tool or function invocations.
- message roleA message role is a field that identifies whether a conversation message contains system instructions, user input, or an assistant response.
- storage abstractionA storage abstraction is an interface or design boundary that separates data-storage operations from the underlying persistence technology.
- ON CONFLICT clauseThe ON CONFLICT clause is a PostgreSQL SQL construct that specifies how an insert should proceed when it violates a uniqueness or exclusion constraint.
- atomicityAtomicity is the database transaction property that requires all operations in a transaction to succeed together or none to take effect.
- database transactionA database transaction is a sequence of database operations treated as one unit of work with a shared commit or rollback outcome.
- psycopgpsycopg2 is a Python adapter library for connecting to and communicating with PostgreSQL databases.
- proto fileA proto file is a Protocol Buffers definition file that declares structured messages and service operations for code generation.
- protocThe Protocol Buffer compiler is a code-generation tool that transforms Protocol Buffer service and message definitions into language-specific source code.
- software development kitA software development kit is a collection of programming tools, libraries, documentation, and related resources for building applications against a software platform or service.
- stubA stub is a generated client-side proxy that represents remote operations as local callable methods.
- token limitA token limit is a configuration constraint specifying the maximum number of model tokens permitted for an input, output, or request.
- PostgreSQLPostgreSQL is an open-source object-relational database management system.
- database indexA database index is a persisted auxiliary structure that accelerates retrieval of rows matching query conditions.
- schemaA database schema is the formal organization of a database's tables, columns, relationships, and constraints.
- context windowA context window is the bounded amount of input and prior interaction content that a language model can process for one generation.
- lost in the middleThe lost in the middle phenomenon is a language-model behavior in which information positioned in the middle of a long context receives less attention than information near the beginning or end.
- summarizationSummarization is the generation of a shorter representation that preserves the salient information from a longer text or interaction.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
