Chapter 5: Enabling conversational memory
This introduction explains why an LLM’s lack of memory makes follow-up conversation difficult.
After this chapter you will be able to
- Stateless LLM Conversation Problem. Explain that an LLM retains no memory after a prompt completes and how that affects a follow-up question that depends on an earlier one.
- Conversation history as context. Trace how Spring AI stores user and assistant messages and uses earlier exchanges from the same conversation to give the LLM context for a new question.
- Role-based and templated history. Distinguish how MessageChatMemoryAdvisor and PromptChatMemoryAdvisor place stored chat history into a prompt, including why the templated approach is useful.
- Context for follow-up questions. Explain why a follow-up question may not make sense without an earlier exchange, using the Burger Force Field example.
- Chat memory advisor replay. Describe how MessageChatMemoryAdvisor and PromptChatMemoryAdvisor provide context for the next question.
- Default advisor for chat memory. Explain how the section enables simple chat memory when creating the ChatClient and identify the chat memory supplied to the advisor.
- Advisor prompt representations. Distinguish how MessageChatMemoryAdvisor and PromptChatMemoryAdvisor send the same stored history to the LLM. Describe what appears in each prompt at the start of a conversation and after a follow-up question.
- Growing history token cost. Explain why sending an increasingly long conversation history in prompts raises token cost and can eventually exceed the context window limit.
- Recent-message memory window. Explain that Spring AI sends only the most recent 20 chat messages by default, so older messages drop out as new ones are added.
- Custom chat memory limit. Describe how to change the message limit by overriding the autoconfigured ChatMemory bean with a MessageWindowChatMemory built using maxMessages(). Explain the tradeoff between more conversational context and greater token usage.
- Conversation identity and isolation. Explain why a multi-user application needs distinct conversation IDs and how the section’s follow-up questions demonstrate their effect on conversational context.
- Conversation ID request path. Trace how the example takes a conversation ID from an HTTP request and supplies it to the chat memory advisors.
- Application-managed memory limits. Explain that a chat memory advisor lets Board Game Buddy use past exchanges without giving the LLM memory of its own, and that its current MessageWindowChatMemory is lost when the application restarts.
- Durable chat memory. Explain why a persistent ChatMemoryRepository lets conversation history survive application restarts while continuing to provide context for later messages.
- Cassandra and Neo4j setup. Describe how the Cassandra and Neo4j starters supply repository beans through autoconfiguration, and what database setup is still required.
- JDBC repository configuration. Explain how JDBC chat memory setup differs from Cassandra and Neo4j, including the explicit repository bean, database dialect, DataSource, and schema initialization.
- Similarity-based chat memory retrieval. Explain how VectorStoreChatMemoryAdvisor stores chat messages as embeddings and retrieves messages similar to the current question for inclusion in the LLM’s system prompt, rather than simply using the most recent messages.
What it builds on
- Assigning prompt roles (chapter 3) · The distinction assumes familiarity with user, assistant, and system prompt roles.
- Understanding RAG (chapter 4) · The comparison to RAG assumes the idea of retrieving similar material for prompt context.
- Implementing RAG with an advisor (chapter 4) · Uses the earlier addition of QuestionAnswerAdvisor as a default advisor.
- Setting up a vector store (chapter 4) · Having a vector store available to the application.
Section by section
Introduction
This introduction explains why an LLM’s lack of memory makes follow-up conversation difficult.
- Stateless LLM Conversation Problem: Explain that an LLM retains no memory after a prompt completes and how that affects a follow-up question that depends on an earlier one.
Why it matters: It explains why a seemingly simple follow-up question can fail in a conversation with an LLM.
Key terms: LLMs, stateless services, conversational state
5.1 Making memories in AI
The section explains how Spring AI preserves conversation history and presents three advisors that add that history to later prompts.
- Conversation history as context: Trace how Spring AI stores user and assistant messages and uses earlier exchanges from the same conversation to give the LLM context for a new question.
Why it matters: Including earlier exchanges gives the model the context it needs to stay on topic. - Role-based and templated history: Distinguish how MessageChatMemoryAdvisor and PromptChatMemoryAdvisor place stored chat history into a prompt, including why the templated approach is useful.
Why it matters: The advisor determines how a model receives the same stored conversation history. - Relevant vector-store history: Explain how VectorStoreChatMemoryAdvisor selects and supplies chat history differently from the two ChatMemory-based advisors.
Why it matters: This advisor brings question-relevant pieces of the conversation into the prompt rather than simply presenting the stored dialog.
Key terms: ChatMemory, user message, assistant message, chat history, prompt context, MessageChatMemoryAdvisor, PromptChatMemoryAdvisor, role-based messages, system message template, VectorStoreChatMemoryAdvisor
5.2 Adding conversational memory
Explains why follow-up questions need conversational context and how Spring AI chat memory advisors supply it.
- Context for follow-up questions: Explain why a follow-up question may not make sense without an earlier exchange, using the Burger Force Field example.
Why it matters: Conversational memory lets the assistant understand what a follow-up refers to. - Chat memory advisor replay: Describe how MessageChatMemoryAdvisor and PromptChatMemoryAdvisor provide context for the next question.
Why it matters: Replaying both sides of the exchange gives the LLM the context needed to respond to a follow-up.
Key terms: memory of the conversation, follow-up questions, MessageChatMemoryAdvisor, PromptChatMemoryAdvisor, LLM
5.2.1 Enabling an in-memory chat advisor
This section shows how one default advisor enables in-memory chat memory in Board Game Buddy, then illustrates the result with a follow-up question.
- Default advisor for chat memory: Explain how the section enables simple chat memory when creating the ChatClient and identify the chat memory supplied to the advisor.
Why it matters: A small ChatClient configuration change lets the application retain conversational context. - Default memory storage lifetime: Describe where the default MessageWindowChatMemory keeps conversation history and what happens to it when the application restarts.
Why it matters: The default is convenient for getting started, but its conversation history is temporary. - Follow-up context as evidence: Explain what the Board Game Buddy follow-up demonstrates about the newly enabled chat memory.
Why it matters: The follow-up shows the practical difference conversational memory makes to an answer.
Key terms: ChatClient, defaultAdvisors(), MessageChatMemoryAdvisor, ChatMemory, MessageWindowChatMemory, PromptChatMemoryAdvisor, ChatMemoryRepository, Map, conversation history, follow-up question
5.2.2 Inspecting the prompt for chat memory
The section shows how two chat memory advisors place conversation history in prompts and why growing history can become costly.
- Chat history message roles: Explain that chat history stores the user's questions as user messages and the LLM's answers as assistant messages, while system messages carry application instructions.
Why it matters: These roles make it easier to recognize what is being carried forward in a prompt. - Advisor prompt representations: Distinguish how MessageChatMemoryAdvisor and PromptChatMemoryAdvisor send the same stored history to the LLM. Describe what appears in each prompt at the start of a conversation and after a follow-up question.
Why it matters: The distinction explains why the two advisors produce different-looking prompts from the same conversation. - Growing history token cost: Explain why sending an increasingly long conversation history in prompts raises token cost and can eventually exceed the context window limit.
Why it matters: It explains why an application may need to limit how much history it sends with each question.
Key terms: chat history, user message, assistant message, system message, MessageChatMemoryAdvisor, PromptChatMemoryAdvisor, chat memory, system prompt, usage tokens, token cost
5.2.3 Configuring chat memory size
This section explains how Spring AI limits chat memory to recent messages and how to change that limit.
- Recent-message memory window: Explain that Spring AI sends only the most recent 20 chat messages by default, so older messages drop out as new ones are added.
Why it matters: It explains why an assistant may lose access to something said early in a long conversation. - Custom chat memory limit: Describe how to change the message limit by overriding the autoconfigured ChatMemory bean with a MessageWindowChatMemory built using maxMessages(). Explain the tradeoff between more conversational context and greater token usage.
Why it matters: It lets you choose how much conversation context to retain without using more tokens than you want.
Key terms: chat memory, messages, prompt, ChatMemory, MessageWindowChatMemory, maxMessages(), @Bean, token usage
5.3 Specifying the conversation ID
This section shows how to give requests distinct conversation IDs so chat memory stays within the right conversation, then explains why the default in-memory storage is unsuitable for production.
- Conversation identity and isolation: Explain why a multi-user application needs distinct conversation IDs and how the section’s follow-up questions demonstrate their effect on conversational context.
Why it matters: The conversation ID determines which history can help answer a follow-up question. - Conversation ID request path: Trace how the example takes a conversation ID from an HTTP request and supplies it to the chat memory advisors.
Why it matters: Passing the ID all the way to the advisors lets them associate a request with the intended conversation. - Default memory production limits: Explain why the chat memory setup used so far is not recommended for production.
Why it matters: A conversation that works during a test may lose its history or use increasing application memory in production.
Key terms: conversation ID, default, conversation history, X_AI_CONVERSATION_ID, @RequestHeader, AdvisorSpec, CONVERSATION_ID, MessageWindowChatMemory, ChatMemoryRepository, in-memory
5.4 Enabling persistent chat memory
Explains why application-managed chat memory enables conversational recall but needs persistence to survive an application restart.
- Application-managed memory limits: Explain that a chat memory advisor lets Board Game Buddy use past exchanges without giving the LLM memory of its own, and that its current MessageWindowChatMemory is lost when the application restarts.
Why it matters: It explains why a conversational application may still need persistent chat memory.
Key terms: chat memory advisor, MessageWindowChatMemory, persistent chat memory
5.4.1 Persisting chat memory to a database
The section shows how to replace in-memory chat storage with Cassandra, Neo4j, or JDBC-backed storage so conversation history survives application restarts.
- Durable chat memory: Explain why a persistent ChatMemoryRepository lets conversation history survive application restarts while continuing to provide context for later messages.
Why it matters: A restarted application can continue a conversation only if its history is stored somewhere durable. - Cassandra and Neo4j setup: Describe how the Cassandra and Neo4j starters supply repository beans through autoconfiguration, and what database setup is still required.
Why it matters: For these two backends, changing the starter and making the database available is largely enough to switch storage. - Neo4j conversation graph: Describe how the section depicts stored Neo4j chat history as conversations, messages, and metadata.
Why it matters: The graph view makes it easier to see how messages belong to a conversation. - JDBC repository configuration: Explain how JDBC chat memory setup differs from Cassandra and Neo4j, including the explicit repository bean, database dialect, DataSource, and schema initialization.
Why it matters: JDBC persistence requires configuration that the Cassandra and Neo4j starters handle automatically.
Key terms: ChatMemoryRepository, MessageWindowChatMemory, persistent chat memory, CassandraChatMemoryRepository, Neo4jChatMemoryRepository, autoconfiguration, starter dependency, session/conversation entity, user and assistant messages, metadata node
5.4.2 Storing chat memory in a vector store
This section explains how VectorStoreChatMemoryAdvisor retrieves relevant past messages from a vector store and how to configure it on a ChatClient.
- Similarity-based chat memory retrieval: Explain how VectorStoreChatMemoryAdvisor stores chat messages as embeddings and retrieves messages similar to the current question for inclusion in the LLM’s system prompt, rather than simply using the most recent messages.
Why it matters: This explains why the memory supplied to the model can be more relevant than a fixed window of recent messages. - Vector store advisor configuration: Describe how to add VectorStoreChatMemoryAdvisor to a ChatClient and what dependency it takes instead of the ChatMemory implementation used by the other memory advisors. State how the section says a conversation ID can still be supplied.
Why it matters: It connects the retrieval approach to the small configuration change needed to use it in an existing ChatClient.
Key terms: VectorStoreChatMemoryAdvisor, vector store, embeddings, similarity search, RAG pattern, system prompt, ChatClient, VectorStore, ChatMemory, CONVERSATION_ID
5.5 Summary
Where this is used later
- Understanding agents (chapter 12) · Memory for conversational context.
- Asking questions about images (chapter 8) · The use of conversational memory to provide chat history.
- Applying per-user conversational memory (chapter 10) · Per-user conversational memory depends on keeping conversations distinct.
- Tracing AI operations (chapter 9) · The use of a vector store for chat memory.
Concepts explained in this chapter
- messagewindowchatmemoryMessageWindowChatMemory is a Spring AI chat-memory implementation that maintains a bounded window of conversational messages through a ChatMemoryRepository.
- chatmemoryrepositoryChatMemoryRepository is an abstraction for storing and retrieving chat history used by chat-memory implementations.
- MessageChatMemoryAdvisorA chat memory advisor is a Spring AI advisor that manages conversational memory by supplying prior dialogue to later language-model prompts.
- CassandraChatMemoryRepositoryCassandraChatMemoryRepository is a Spring AI ChatMemoryRepository implementation that persists chat history in Apache Cassandra.
- JdbcChatMemoryRepositoryJdbcChatMemoryRepository is a Spring AI ChatMemoryRepository implementation that persists chat history through JDBC in a relational database.
- Neo4jChatMemoryRepositoryNeo4jChatMemoryRepository is a Spring AI ChatMemoryRepository implementation that persists chat history in Neo4j.
- VectorStoreChatMemoryAdvisorVectorStoreChatMemoryAdvisor is a Spring AI advisor that stores chat messages as embeddings in a vector store and retrieves relevant messages as conversational context.
- advisorCallAdvisor is a Spring AI advisor interface for inspecting and modifying a language-model request before execution and its response before return.
- language modelA large language model is a machine-learning model trained on extensive text data to understand and generate natural-language content.
- Postgres driverThe PostgreSQL JDBC driver is a Java library that enables JDBC applications to connect to PostgreSQL databases.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.