liveBook · Manning

Session service

As explained in Designing AI Systems

A session service is a platform service that stores and manages conversational state, user preferences, interactions, and conversation history across sessions.

A session service provides conversation memory for AI applications. It tracks interactions, preferences, and history so an assistant can interpret follow-up questions rather than treating every model call as a fresh conversation. As a shared platform service, it avoids rebuilding session management for each application. section 1.4 section 1.7

Sessions are structured around users, conversations, and ordered messages. The service persists history, while a workflow retrieves the context needed for a model call and records the resulting exchange. The Data Service complements it by retrieving organizational knowledge rather than conversational state. section 4.0 section 1.4

The service contract exposes five core operations:

  • Get or create a session for a user, optionally using an existing session ID. section 4.2
  • List a user's sessions so a client can select one to resume. section 4.2
  • Add messages and retrieve conversation history; retrieval supports limits, offsets, and a strategy parameter. section 4.2
  • Delete a session when its conversation is no longer needed. section 4.2
Listing 4.3 Protocol Buffer service definition for the Session Service
syntax = "proto3";
 
package sessions;
 
service SessionService {
    rpc GetOrCreateSession(GetOrCreateSessionRequest) 
        returns (GetOrCreateSessionResponse);
    rpc ListSessions(ListSessionsRequest)
        returns (ListSessionsResponse);
    rpc AddMessages(AddMessagesRequest) 
        returns (AddMessagesResponse);
    rpc GetMessages(GetMessagesRequest) 
        returns (GetMessagesResponse);
    rpc DeleteSession(DeleteSessionRequest) 
        returns (DeleteSessionResponse);
}
 

The Protocol Buffer contract groups the five operations behind a single gRPC service. section 4.2

Conversation memory in a patient workflow

For a first question, the workflow gets or creates a session and loads its ordered history. It adds the current question, calls the model, and persists both the question and response. A subsequent question such as “What about for my child?” can then refer back to the documents discussed previously. section 2.1.2

Listing 2.2 Adding conversational memory through Session Service
def handle_patient_conversation(question, patient_id, session_id=None):
    session = platform.sessions.get_or_create(patient_id, session_id)    #A
    history, _ = platform.sessions.get_messages(session.session_id)    #B
    messages = [{"role": m.role, "content": m.content} for m in history]    #B
    messages.append({"role": "user", "content": question})    #B
    response = platform.models.chat(    #C
        model="gpt-4o",    #C
        system_prompt_name="patient_intake_assistant",    #C
        messages=messages,    #C
    )
    platform.sessions.add_messages(    #D
        session_id=session.session_id,    #D
        messages=[    #D
            {"role": "user", "content": question},    #D
            {"role": "assistant", "content": response.content},    #D
        ],    #D
    )    #D
    return response.content

The SDK example shows session retrieval, history assembly, the model call, and message persistence in sequence. section 2.1.2

Persistent history and model context are not the same thing. A long conversation can exceed the model's available window, especially when prompts, tools, and retrieved documents also consume space. A sliding window selects recent messages while retaining older ones in storage, but it can leave important early details out of the model request. Reusable strategies let workflows choose an approach suited to the application. section 4.7 section 10.3

Model-managed memory complements message history by storing selected facts across sessions. For example, a workflow can load a patient's saved allergy and medication facts into the prompt, then execute model-issued tool calls to save new facts. Structured memories retain key information even when the original messages are summarized or omitted; they do not replace the full conversational record. section 4.6.5 section 4.7

A production service must also handle concurrent access, such as a user continuing a conversation across tabs or devices. A PostgreSQL backend provides persistent storage, while the SDK and API gateway connect applications to the service's gRPC interface. section 4.0 section 10.3

How Designing AI Systems covers it

How deep it goes
Defines a five-operation contract and develops storage, model-managed memory, and context-window strategies.
Examples
Uses patient intake follow-ups, persistent allergy and medication facts, and a restaurant assistant to illustrate conversational continuity.
What you can do after
Apply it via the platform SDK to persist and retrieve conversational context around model calls.
How it connects
Connects conversation memory with workflow orchestration, semantic search, model calls, and context-window management.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Designing AI Systems. The section links point to the parts of the book it draws on.