liveBook · Manning

Chapter 4: The Session Service: Teaching your AI to remember

What you will learn in chapter 4 of Designing AI Systems

This chapter introduces the Session Service as the platform component that preserves conversational context, stores it as structured session data, and prepares it within model context-window limits.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This chapter introduces the Session Service as the platform component that preserves conversational context, stores it as structured session data, and prepares it within model context-window limits.

Key terms: Session Service, conversation memory, stateless AI system, context-aware intelligence, sessions, conversations, messages, roles, content, session ID

4.1 What a session contains

Defines the lightweight session and message data model that preserves conversational context, including tool-use exchanges, for efficient AI memory.

Key terms: Message, role, content, user, assistant, system, tool_calls, tool_call_id, tool, Function

4.2 The session service contract

Defines the gRPC and Protocol Buffers contract through which workflows create or resume sessions, manage their lifecycle, and record or retrieve conversation history.

Key terms: GetOrCreateSession, ListSessions, DeleteSession, AddMessages, GetMessages, session management, GetOrCreateSessionRequest, GetOrCreateSessionResponse, Session, user ID

4.3 Storage abstraction

This section explains why the Session Service separates session logic from persistence through a pluggable storage abstraction so teams can choose databases that fit their requirements.

Key terms: storage abstraction, storage interface, pluggable storage layer, persistence, Session Service, ACID compliance, audit trails, durability, TTL, PostgreSQL

4.3.1 Choosing the right database

This section compares session-data storage backends, explains how to choose among their tradeoffs, and establishes PostgreSQL as the book’s implementation example.

Key terms: relational (SQL) databases, PostgreSQL, in-memory stores, Redis, document databases, MongoDB, storage abstraction, compliance needs, query patterns, team expertise

4.3.2 The storage interface

This section defines a minimal storage abstraction that lets the Session Service use interchangeable persistence backends without changing its service or application logic.

Key terms: SessionStorage, abstract base class, session persistence, message history, gRPC operations, PostgreSQL, Redis, DynamoDB, SQL queries, in-memory implementation

4.4 Implementing the Session Service backend

This section introduces the PostgreSQL-backed Session Service implementation and the storage-abstraction patterns it demonstrates.

Key terms: PostgreSQL, SessionStorage, storage interface, Session Service, database operations, error handling, connection management, SQL queries, storage implementation

4.4.1 The Database Schema

This section designs a PostgreSQL schema that stores conversation metadata and message history efficiently while preserving their relationship.

Key terms: sessions table, messages table, session ID, user ID, foreign key, primary key, index, CREATE INDEX, CREATE INDEX CONCURRENTLY, SERIAL

4.4.2 The Storage Implementation

This section implements PostgreSQL-backed session storage, covering session creation, atomic message writes, paginated retrieval, lifecycle operations, and row-to-object conversion.

Key terms: SessionStorage, psycopg2, connection string, cursor, SQL queries, storage layer, get_or_create_session, session ID, user ID, UUID

4.4.3 The gRPC service implementation

This section implements the Session Service gRPC layer as a thin, dependency-injected bridge between API Gateway Protocol Buffer requests and storage operations.

Key terms: gRPC service, Protocol Buffer requests, Protocol Buffer responses, SessionServiceServicer, SessionStorage, platform.sessions.get_or_create(), API Gateway, x-target-service, Session Service container, Protocol Buffer request

4.5 Integrating with the SDK

This section integrates the Session Service into the SDK so developers can use simple Python calls that are translated into routed gRPC requests and Python session objects.

Key terms: SessionClient, SessionServiceServicer, BaseClient, GenAIPlatform, API gateway, routing metadata, proto file, Protocol Buffer compiler, sessions_pb2.py, sessions_pb2_grpc.py

4.6 Model-managed memory

This section introduces model-managed memory, where an assistant explicitly saves important facts outside the conversation history so they persist across long interactions.

Key terms: model-managed memory, LLM, context window, external storage, RAM, disk, save_memory tool, memory store, structured data, conversation history

4.6.1 Where memory lives

Defines how memories are associated with users and optionally sessions so applications can support temporary context, persistent facts, provenance, and application-specific deletion policies.

Key terms: user ID, session ID, session-scoped memory, user-level persistence, memory model, provenance, debugging, auditability, cascade, session lifecycle

4.6.2 The memory model

This section defines the structure and update behavior of model-managed memory entries, including strategies for concurrent changes to user-scoped facts.

Key terms: MemoryEntry, key, value, session_id, updated_at, append-only log, memories, overwrite, last-write-wins, optimistic concurrency control

4.6.3 Extending the storage interface for memories

This section extends the SessionStorage abstraction with consistent memory persistence operations, user/session scoping, privacy deletion, and backend-independent implementations.

Key terms: SessionStorage, save_memory, get_memory, delete_memory, clear_user_memory, user_id, session_id, session-scoped memory, user-level persistence, provenance

4.6.4 SDK Integration

This section integrates model-managed memory into the SDK by exposing SessionClient methods that serialize flexible memory values for gRPC transport and return native Python objects to workflows.

Key terms: SessionClient, Session Service, save_memory, get_memory, delete_memory, clear_user_memory, JSON encoding, JSON decoding, MemoryEntry, gRPC

4.6.5 A complete workflow example

This section demonstrates a healthcare-assistant workflow that loads persistent memories, lets the model save important new facts, and keeps conversation history separate from durable knowledge.

Key terms: model-managed memory, save_memory, Session Service, conversation turn, get_memory, system prompt, memories, session, messages, conversation history

4.7 Managing context windows

This introduction frames context engineering as the problem of selecting the right session information for a limited model context window, with the platform supplying flexible strategies and workflows choosing how to use them.

Key terms: context window, system prompts, tool definitions, retrieved documents, token budget, Session Service, platform, workflows, building blocks, reusable strategies

4.7.1 The token budget problem

This section frames the context window as a shared token budget and explains why the Session Service must choose which conversation messages to include.

Key terms: context window, token budget, conversation history, system prompt, tool definitions, retrieval-augmented generation (RAG), Session Service, message selection

4.7.2 Simple truncation

This section explains simple truncation as a fast way to fit conversation history into a token budget, while showing why it can lose important older context.

Key terms: simple truncation, token budget, context window, LLM calls, safety-critical information

4.7.3 Compressing history with summarization

This subsection explains how LLM-generated summaries compress older conversation history to extend effective context while managing fidelity, cost, latency, and freshness.

Key terms: summarization, effective context, verbatim messages, truncation, token count, summarization prompt, generic summaries, medical conditions, allergies, safety concerns

4.7.4 Hierarchical memory

This section explains how hierarchical memory combines durable facts, compressed history, and verbatim recent messages while allocating and ordering context to preserve what models need most.

Key terms: hierarchical memory, model-managed memories, rolling summary, verbatim recent messages, conversational fidelity, lost in the middle, context assembly, critical facts, safety information, token budget

4.7.5 Retrieval-augmented memory

Retrieval-augmented memory extends an assistant beyond the current session by searching a user’s past conversations and adding relevant results to the current context.

Key terms: retrieval-augmented memory, past sessions, current conversation, conversation history, Search Service, embeddings, semantic search, search query, ranked and filtered, token budget

4.7.6 Putting it together

This section shows how applications choose and combine context-management strategies based on user behavior, safety needs, token budgets, and cross-session recall requirements.

Key terms: simple truncation, summarization, model-managed memory, retrieval-augmented memory, token budget, GetMessagesRequest, strategy field, Session Service, Data Service, hierarchical strategy

4.8 Summary

This summary recaps how the Session Service preserves conversational state, supports durable memory, and manages limited model context windows.

Where this is used later

Concepts explained in this chapter

← Chapter 3: The Model Service: Your platform’s gateway to AI models Chapter 5: The Data Service: teaching AI what your organization knows →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.