Chapter 11: Managing session state and chat history
The introduction identifies the context-window limit as a reason a long chatbot conversation may become slower and less coherent.
After this chapter you will be able to
- Long conversation degradation. Explain that a chatbot may repeat itself, contradict earlier statements, and respond more slowly during a long conversation because it has hit the context-window limit—not necessarily because it is broken.
- Stateless LLM requests. Explain that each ollama.chat() call is independent and the app must send relevant earlier messages with each new request for the model to respond as if it remembers the conversation.
- Context window limit. Explain that the model’s context window limits how much conversation history fits in one request, so a long chat cannot always be sent in full.
- Application versus model memory. Distinguish storing visible messages in Streamlit session state from sending messages to the LLM, and explain that session state does not cause the model’s context-window limit.
- Reruns and persistent chat history. Explain why ordinary Python variables cannot reliably hold a Streamlit chat history and how st.session_state solves that problem.
- One-time session state initialization. Explain why the counter is initialized only on the first app load and how the same pattern preserves the chatbot's message list.
- Values that persist across reruns. Explain that session state can hold almost any Python object and that a value needed after an app reruns should be placed in st.session_state.
- Session State Reset Conditions. State when session state persists and identify the actions that reset it: closing the browser tab, refreshing the page, or stopping and restarting the Streamlit server.
- Conversation history and apparent memory. Explain how the chatbot gives a stateless LLM conversational context by passing st.session_state.messages to ollama.chat() on every request.
- Growing history and token cost. Explain why longer conversations slow responses and increase the chance of reaching the context limit, and state the section’s rough English token estimate.
- Context window capacity. Explain that a context window is the maximum text an LLM can process at one time, measured in tokens, and that models differ in capacity. Distinguish a model’s capacity from the active context length allocated by Ollama.
- Conversation token budget. Explain that the context window must fit the user’s messages, the AI’s replies, and the new reply being generated, so a long conversation can consume it quickly.
- Context window overflow effects. Explain what can happen when a conversation exceeds a model’s context window: earlier messages may be silently dropped, or the model may give confused or contradictory responses.
- Recent history sent to model. Explain how the chatbot limits the history sent to the model without removing older messages from the browser, and describe what max_history counts.
- Limited history tradeoff. Explain that trimming chat history removes earlier context from the model, but that keeping a limited history is sufficient for most conversations.
- Clear Chat confirmation flow. Trace how the two-step Clear Chat flow uses session state, and explain how confirmation and cancellation affect the chat history.
- Persistent conversation start time. Explain why the application creates a conversation start timestamp only if it is absent from session_state, and how it uses that timestamp.
- Selected model change detection. Describe how comparing the selected model with the model stored in session_state lets the application detect a change and respond to it.
- Message and history flow. Trace what happens after a user submits a chat message: Streamlit reruns, preserves and displays existing messages, adds the new message, and sends trimmed history to Ollama.
- Streaming response and storage. Trace how the local LLM's response appears and is saved, and explain why streaming improves the perceived wait.
What it builds on
- The LLM’s memory problem (chapter 7) · Introduces the LLM’s memory problem.
- Creating a virtual environment (chapter 6) · Creating the virtual environment used to run the examples.
- How Streamlit’s execution model works (chapter 8) · Introduces Streamlit’s script execution model.
- Starting and stopping (chapter 8) · Starting a Streamlit app.
- The sidebar: Controlling two models (chapter 10) · The application has a model-selection control.
- The streaming communication flow (chapter 7) · the streaming communication flow
Section by section
Introduction
The introduction identifies the context-window limit as a reason a long chatbot conversation may become slower and less coherent.
- Long conversation degradation: Explain that a chatbot may repeat itself, contradict earlier statements, and respond more slowly during a long conversation because it has hit the context-window limit—not necessarily because it is broken.
Why it matters: Recognizing these symptoms helps you identify the problem the chapter addresses.
Key terms: context window limit
11.1 The problem: LLMs are stateless
This section explains why a chat app must retain and resend conversation history while keeping that history within the model’s context window.
- Stateless LLM requests: Explain that each ollama.chat() call is independent and the app must send relevant earlier messages with each new request for the model to respond as if it remembers the conversation.
Why it matters: This explains why a follow-up question needs earlier messages sent along with it. - Context window limit: Explain that the model’s context window limits how much conversation history fits in one request, so a long chat cannot always be sent in full.
Why it matters: Keeping history is useful only if the app also considers how much the model can process at once. - Application versus model memory: Distinguish storing visible messages in Streamlit session state from sending messages to the LLM, and explain that session state does not cause the model’s context-window limit.
Why it matters: The app can keep a visible chat record without assuming the model has read or retained all of it.
Key terms: stateless, ollama.chat(), conversation history, context window, st.session_state, application memory, model memory, visible chat history
11.2 Understanding Streamlit session state
This section introduces Streamlit session state and gives setup instructions for running the chapter’s examples.
- Chapter Example Setup: Describe how to save and run the chapter’s Streamlit examples, and what to check if Streamlit reports a missing module.
Why it matters: Matching the file, environment, and interpreter helps the chapter’s examples run as intended.
Key terms: chatbot.py, streamlit run chatbot.py, virtual environment, interpreter
11.2.1 The script rerun problem
This section explains why Streamlit chat apps need session state to keep conversation history across user interactions.
- Reruns and persistent chat history: Explain why ordinary Python variables cannot reliably hold a Streamlit chat history and how st.session_state solves that problem.
Why it matters: Keeping history across reruns lets a chat app retain earlier messages when the user sends a new one.
Key terms: script rerun, regular Python variable, conversation history, session state, st.session_state
11.2.2 Session state in action
A counter demo shows how Streamlit session state keeps a value across script reruns and how the same initialization pattern preserves chatbot messages.
- One-time session state initialization: Explain why the counter is initialized only on the first app load and how the same pattern preserves the chatbot's message list.
Why it matters: This pattern lets a rerunning Streamlit app retain the conversation it has already built. - Running the counter demo: Describe how to launch the counter demo and what its buttons demonstrate.
Why it matters: Running the demo makes session-state persistence visible when you click its buttons.
Key terms: st.session_state, initialization check, messages, counter_demo.py, streamlit run, Increment, Decrement
11.2.3 What you can store in session state
This subsection gives a rule for deciding which values to keep in Streamlit session state across app reruns.
- Values that persist across reruns: Explain that session state can hold almost any Python object and that a value needed after an app reruns should be placed in st.session_state.
Why it matters: This rule helps you decide which app values must survive a user interaction.
Key terms: session state, st.session_state, reruns
11.2.4 When session state resets
This section explains when Streamlit session state resets and why separate users can have independent conversations.
- Session State Reset Conditions: State when session state persists and identify the actions that reset it: closing the browser tab, refreshing the page, or stopping and restarting the Streamlit server.
Why it matters: It explains why a conversation may disappear even when the app is still available. - Independent User Sessions: Explain why multiple users of a deployed application can have separate conversations.
Why it matters: Separate session state keeps users' conversations from mixing in a deployed app.
Key terms: session state, browser tab, Streamlit server, independent session state, separate conversations
11.3 How LLMs use conversation history
This section explains how a stateless LLM appears to remember a chat because the app resends its message history, and why that becomes costly as the history grows.
- Conversation history and apparent memory: Explain how the chatbot gives a stateless LLM conversational context by passing st.session_state.messages to ollama.chat() on every request.
Why it matters: This explains why the chatbot can refer to earlier turns even though the model itself does not remember them. - In-place history updates: Describe why appending a message to st.session_state.messages makes it part of the history available for the next model request without a separate synchronization step.
Why it matters: It connects a new chat message in session state to what the model receives next. - Growing history and token cost: Explain why longer conversations slow responses and increase the chance of reaching the context limit, and state the section’s rough English token estimate.
Why it matters: This is the reason managing retained chat history matters as a conversation gets longer.
Key terms: stateless, ollama.chat(), messages=st.session_state.messages, conversation history, Python lists, mutable, st.session_state.messages, appending, retained conversation history, tokens
11.4 Context window management
This section explains how a model’s token-limited context window constrains the amount of conversation it can process at once.
- Context window capacity: Explain that a context window is the maximum text an LLM can process at one time, measured in tokens, and that models differ in capacity. Distinguish a model’s capacity from the active context length allocated by Ollama.
Why it matters: A model’s advertised limit may differ from the context actually available in your running setup. - Conversation token budget: Explain that the context window must fit the user’s messages, the AI’s replies, and the new reply being generated, so a long conversation can consume it quickly.
Why it matters: Counting only the latest message understates how much of the available context a chat uses.
Key terms: context window, tokens, model capacities, active context, ollama ps, conversation history
11.4.1 What happens when you exceed the context window
This subsection explains how an overlong conversation can degrade a chatbot’s responses and why sending more context also slows it down.
- Context window overflow effects: Explain what can happen when a conversation exceeds a model’s context window: earlier messages may be silently dropped, or the model may give confused or contradictory responses.
Why it matters: It helps explain why a long-running chatbot may suddenly seem to forget or contradict itself. - Growing context performance cost: Explain why context window management matters even for a model with a large window: sending more data increases processing time for every response.
Why it matters: It shows why staying under a large context limit is not the only consideration when managing chat history.
Key terms: context window, earliest messages, response quality, large context windows, processing time, performance
11.4.2 Implementing history trimming
This section shows how to send only recent chat history to the model while keeping the full conversation visible to the user.
- Recent history sent to model: Explain how the chatbot limits the history sent to the model without removing older messages from the browser, and describe what max_history counts.
Why it matters: It lets a chat app preserve the visible conversation while reducing what it sends to the model. - Sidebar conversation size statistics: Describe which conversation statistics the sidebar displays and what they help the user monitor.
Why it matters: These figures give the user a way to notice a growing conversation.
Key terms: max_history, completed exchanges, context window, message-count trimming, conversation statistics, total number of messages, approximate token count
11.4.3 The limited history tradeoff
This section explains the context lost when chat history is trimmed, why trimming is usually sufficient, and two ways to retain access to older information.
- Limited history tradeoff: Explain that trimming chat history removes earlier context from the model, but that keeping a limited history is sufficient for most conversations.
Why it matters: This helps you decide whether losing older messages is an acceptable cost of limiting chat history. - Older context alternatives: Distinguish summarization from RAG as two approaches to retaining access to older conversation information.
Why it matters: These approaches offer ways to bring older information back when simple trimming is not enough.
Key terms: history trimming, context, exchanges, Summarization, RAG, retrieval-augmented generation
11.5 Adding a reset button
This section adds a session-state confirmation step to Clear Chat so a user cannot erase a conversation with one click.
- Clear Chat confirmation flow: Trace how the two-step Clear Chat flow uses session state, and explain how confirmation and cancellation affect the chat history.
Why it matters: A confirmation step lets users back out before losing their conversation.
Key terms: Clear Chat, confirm_clear, session state, messages
11.6 Other useful session state patterns
The section shows two session-state enhancements: preserving a conversation start time across Streamlit reruns and detecting when the user changes models.
- Persistent conversation start time: Explain why the application creates a conversation start timestamp only if it is absent from session_state, and how it uses that timestamp.
Why it matters: A stable timestamp makes conversation duration meaningful across interactions. - Selected model change detection: Describe how comparing the selected model with the model stored in session_state lets the application detect a change and respond to it.
Why it matters: Detecting a model switch gives the application a chance to handle the existing conversation state appropriately.
Key terms: conversation start time, session_state, model-change detection, model-selection widget, conversation state
11.7 The complete chat application data flow
This section traces a chat message through Streamlit, session state, Ollama, and the local LLM, then explains why streaming makes the response feel faster.
- Message and history flow: Trace what happens after a user submits a chat message: Streamlit reruns, preserves and displays existing messages, adds the new message, and sends trimmed history to Ollama.
Why it matters: This flow explains how the chat keeps its conversation despite Streamlit rerunning the script. - Streaming response and storage: Trace how the local LLM's response appears and is saved, and explain why streaming improves the perceived wait.
Why it matters: It distinguishes seeing a response promptly from the LLM actually finishing its generation sooner.
Key terms: st.chat_input(), session state, st.session_state.messages, max_history, ollama.chat(), local LLM, tokens, streaming, placeholder
11.8 Exercises
11.9 Summary
Where this is used later
- The enhanced chatbot (chapter 13) · the use of conversation history in LLM requests
- Key Modelfile instructions (chapter 13) · Conversation history may be trimmed before it is sent to the model.
- What is RAG? (chapter 16) · Introduces RAG as a way to retrieve relevant stored information.
- Step 3: Add a Model Selector in the Streamlit UI (chapter 12) · the existing chat application to which the selector is added
- Final thoughts (chapter 18) · The chat application data flow provides an example of the recurring patterns.
Concepts explained in this chapter
- session contextSession state is application data associated with a user's session and retained across interactions or reruns.
- large language modelA machine-learning model trained on large amounts of text to process and generate language.
- context windowContext length is the maximum number of tokens a language model can consider at once.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.