Chapter 7: Controlling LLMs with the Ollama Python library

Introduction

The chapter introduces Python control of a local LLM and gives the checks needed before running its scripts.

• Programmatic local model control: Explain that a Python script can send instructions to a locally running LLM and receive its responses, rather than only interacting with AI as a user.
  Why it matters: This is the shift from using a local model to controlling it in your own programs.
• Script startup prerequisites: State the setup checks before beginning: activate the virtual environment, have the Ollama service running, and pull gemma3:4b if it is not already downloaded. Explain what the port-11434-in-use message means in this context.
  Why it matters: These checks help you start the chapter's scripts with the required local setup in place.

Key terms: Python scripts, LLM, programmatic control, locally running AI model, virtual environment, (.venv), Ollama service, gemma3:4b, ollama pull gemma3:4b, port 11434

7.1 Your first Ollama Python script

Introduces an upcoming Python script that will ask Gemma 3 a question and print its answer, with the code explained afterward.

7.1.1 The simplest AI program

This subsection has the reader run a small Python script that sends a message to a locally running Gemma 3 model and prints its reply.

• First local AI script: Describe creating and running chat_basic.py, then explain that it sends a message to Gemma 3 on the laptop, receives a reply, and prints it without an internet connection, API key, or cloud service.
  Why it matters: This is the first working example of using a local model from a Python program.

Key terms: chat_basic.py, Gemma 3 model, local

7.1.2 Breaking down the code

This section explains how a short Python script imports Ollama, sends messages to a local model, and prints the model’s reply.

• Ollama library import: Explain that importing Ollama makes the installed Python library’s functions available to the script, and describe a function as named code called with parentheses.
  Why it matters: The import is what lets the script call Ollama from Python.
• Chat call and response flow: Describe the two arguments to ollama.chat() and trace how the messages reach the local model and a response comes back.
  Why it matters: This call is how the Python script gets an answer from a local model.
• Nested reply text extraction: Explain how the script finds the assistant’s reply in the returned dictionary and prints it to the terminal.
  Why it matters: The script must select the reply text from the larger response before displaying it.

Key terms: Ollama Python library, function, import, ollama.chat(), arguments, model, messages, Ollama server, Python dictionary, nested dictionary

7.2 Understanding the message format

This introduction notes that the previous script used a user message and previews three message roles that affect AI behavior.

7.2.1 The three LLM message roles

This subsection explains the three message roles sent to an LLM and what each contributes to a conversation.

• Three LLM message roles: Distinguish system, user, and assistant messages by what each contains and does in a conversation.
  Why it matters: The roles tell the model which text sets its behavior, which comes from the human, and which is its own earlier output.

Key terms: system, user, assistant, role

7.2.2 Adding a system message

This subsection shows that changing a system message can steer the same model’s response style and role.

• System message behavior control: Explain how a system message can steer the same model’s behavior, using the example of an answer that was short and used an analogy as instructed.
  Why it matters: It gives you a way to change how the model responds without changing the model itself.

Key terms: system message

7.2.3 Experimenting with different system messages

Changing the system message can change an LLM’s answer to the same question, though models vary in how closely they follow it.

• System message influence: Explain how replacing the system message while asking the same question demonstrates the system role’s influence on the response.
  Why it matters: It shows how you can change an answer without changing the user’s question.
• Model instruction-following differences: Describe how model size can affect responses to system messages and what to try if a small model ignores one.
  Why it matters: It gives you practical options when a system message has little effect.

Key terms: system message, system role, system instructions, larger models, smaller ones

7.3 Streaming responses for real-time output

Introduces streaming as a change in how a model response reaches a program, not a change to the message roles.

• Streaming and message roles: Distinguish streaming from the earlier scripts that wait for a complete response, and explain that streaming changes how the model’s reply is delivered without adding a message role.
  Why it matters: This distinction helps you understand streaming without treating it as a new kind of message.

Key terms: streaming, system, user, assistant

7.3.1 The problem with waiting for the model’s response

This subsection contrasts waiting for a complete response with streaming text as it appears, explaining why streaming feels more responsive.

• Streaming and perceived responsiveness: Explain why showing a model’s response as it appears can feel better than waiting for the entire answer.
  Why it matters: Streaming improves the experience of waiting for a longer answer.

Key terms: streaming

7.3.2 Implementing streaming

The section has the reader run a streaming chat script and observe its response appear progressively rather than all at once.

• Visible streaming response: Describe how the output looks when the streaming script runs, compared with a response that appears after a pause.
  Why it matters: This visible difference is what the reader can verify when trying the streaming script.

Key terms: streaming, chat_stream.py

7.3.3 Understanding the code

Explains how the streaming Ollama script receives a response piece by piece and prints it as it arrives.

• Streaming response iteration: Trace how `stream=True` changes the result of `ollama.chat()` and how the `for` loop processes it.
  Why it matters: This explains why the script can handle an answer while the model is still generating it.
• Immediate continuous terminal output: Explain how the `print()` settings display incoming chunks continuously and why a final empty `print()` follows them.
  Why it matters: These print settings make the streamed answer readable as it arrives and leave the terminal ready for the next command.

Key terms: stream=True, ollama.chat(), iterator, chunk, for chunk in stream, print(), end="", flush=True, newline, buffering

7.3.4 The streaming communication flow

This section shows how a Python script, the Ollama server, and an LLM communicate during streaming, and why streaming feels faster without necessarily speeding up generation.

• Local AI Communication Path: Describe how the Python script communicates with the Ollama server, which manages the LLM.
  Why it matters: This identifies which component your Python code communicates with.
• Streaming Output and Responsiveness: Explain that streaming prints each token as it arrives, giving the user immediate visible progress without necessarily making the model generate text faster.
  Why it matters: You can explain why streaming improves the experience without claiming it speeds up the model.

Key terms: Python script, Ollama server, LLM model, streaming, token, responsive

7.4 Maintaining conversation history

This introduction contrasts a one-message script with a chatbot that responds to follow-up questions in context.

• Single Replies Versus Conversation: Distinguish a script that sends one message and receives one response from a chatbot that supports follow-up questions and responds in context.
  Why it matters: This distinction explains why a one-off AI script does not yet feel like a conversation.

Key terms: single message, single response, follow-up questions, in context

7.4.1 The LLM’s memory problem

Explains why an Ollama chat appears to remember earlier messages and why a Python script must manage conversation history itself.

• Apparent LLM Conversation Memory: Explain that an LLM does not remember earlier messages between requests: Ollama's interactive chat appears to remember because it resends the conversation history each time. Describe why a script must manage that history itself.
  Why it matters: This explains why a custom chat script needs to keep track of earlier messages if replies are to reflect them.

Key terms: LLMs, ollama run, conversation history

7.4.2 Building a conversation loop

This section explains how a Python script repeats a chat exchange while retaining conversation history until the user quits.

• Conversation loop termination: Explain how `while True` keeps the chat script running and how typing `quit` causes a `break` statement to stop the loop.
  Why it matters: This explains why the script can handle multiple turns without running forever after the user wants to leave.
• Conversation history exchange: Trace one pass through the chat script: begin with a system message, get user input, send the full conversation history to the model, display its streamed response, and save both sides of the exchange before repeating.
  Why it matters: Saving both sides of each exchange lets the next model request include the prior conversation.

Key terms: while loop, while True, break, quit, system message, conversation history, streamed response, history list

7.4.3 Running the conversation

This subsection shows how to test a multiturn chat script, why it appears to remember a name, and how to end the chat.

• Name recall through history: Explain why the model can use a name introduced earlier in the conversation: the script sends the full history with each message.
  Why it matters: This explains the apparent memory you see when testing the chat.
• Ending the conversation loop: State how to stop the running chat: type quit, exit, or bye.
  Why it matters: You need a way to leave the chat when you finish testing it.

Key terms: multiturn conversation, full history, infinite loop, quit, exit, bye

7.4.4 How the history list grows

This section traces how a conversation loop adds messages to history and sends that history to Ollama for each reply.

• Conversation History Growth and Flow: Trace one exchange through the history list: add the user message, send the full history to ollama.chat(), stream and collect the response, then add the assistant message. Explain why the list grows by two messages per exchange and how sending it gives the model conversational context.
  Why it matters: The order of these steps explains how the next reply can take earlier exchanges into account.

Key terms: history list, role: "user", role: "assistant", ollama.chat(), response_text

7.4.5 The complete conversation flow

This section summarizes the repeating conversation-history loop behind a chatbot and explains the limit imposed by a model’s context window.

• Chatbot conversation loop: Trace how a chatbot sends conversation history to the model, adds the model’s response to that history, and repeats the process for the next message.
  Why it matters: This loop explains how a chatbot can respond in the context of an ongoing conversation.
• Context window limit: Explain what a context window limits and what happens when conversation history exceeds it.
  Why it matters: Sending the full history works only while it fits within what the model can consider.

Key terms: conversation history, conversation loop, context window

7.5 Exercises

7.6 Summary