Chapter 3: Working with LLMs
This introduction previews adding an LLM abstraction, an Ollama integration, and a complete tool-calling demonstration to the framework.
After this chapter you will be able to
- Common asynchronous LLM interface. Explain why BaseLLM provides a common interface across LLM providers and what async-first means.
- Prompt completion and result. Describe the input and output of BaseLLM's complete() interaction.
- Chat messages and history. Trace the inputs and returned messages of chat(), including where tool-call requests and conversation history fit.
- Completion Result Structure. Describe what CompleteResult stores after a text completion.
- Chat Message Roles. Explain the purpose of the SYSTEM, USER, ASSISTANT, and TOOL roles in chat messages.
- BaseLLM abstract interface. Describe the BaseLLM interface and the responsibility of its subclasses.
- Tool result chat continuation. Trace a tool request through execution and back into the chat using continue_chat_with_tool_results().
- Structured output processing rationale. Explain why structured output is easier to use downstream than free-form LLM responses.
- Structured output model contract. Describe the inputs and return type of structured_output() and the role of the Pydantic model.
- Ollama integration role. Explain how OllamaLLM integrates Ollama-supported models into the framework.
- Completion interface integration. Trace how OllamaLLM implements complete() using Ollama’s completion interface.
- Structured output integration. Trace how structured_output() requests and returns an instance of the desired Pydantic model.
- Chat message and tool conversion. Trace how chat() converts framework inputs and tools for Ollama and returns framework messages.
- Tool result chat continuation. Trace how continue_chat_with_tool_results() accepts tool results and continues an Ollama chat.
- Hailstone tool calling process. Trace the Hailstone tool-calling process using OllamaLLM, from the initial chat request through tool execution and the continued response.
- Backbone LLM selection. Explain how choosing backbone LLMs for different agents affects a multi-agent system’s effectiveness and efficiency, using the coordinator and subordinate example.
- LLM reuse across agents. Explain why the framework’s existing LLM work is sufficient for both a single agent and a multi-agent system.
What it builds on
- The basics of UML diagrams (chapter 1) · Introduces the basics of UML diagrams.
- Implementing ToolCall and ToolCallResult (chapter 2) · The ToolCall data structure.
- Defining BaseTool (chapter 2) · The abstract BaseTool interface and subclass responsibilities.
- Implementing SimpleFunctionTool (chapter 2) · The Hailstone tool created in the previous chapter.
Section by section
Introduction
This introduction previews adding an LLM abstraction, an Ollama integration, and a complete tool-calling demonstration to the framework.
3.1 BaseLLM: a blueprint for LLMs
This section outlines the async BaseLLM interface, its interaction modes, and the data structures that standardize LLM access.
- Common asynchronous LLM interface: Explain why BaseLLM provides a common interface across LLM providers and what async-first means.
Why it matters: A shared asynchronous interface lets the framework work with different LLM providers consistently. - Prompt completion and result: Describe the input and output of BaseLLM's complete() interaction.
Why it matters: This gives the framework a standard way to request and package a text completion. - Chat messages and history: Trace the inputs and returned messages of chat(), including where tool-call requests and conversation history fit.
Why it matters: The returned messages preserve the conversation and carry any tool requests forward. - Message roles and composition: Explain how ChatRole identifies message senders and how its relationship to ChatMessage is depicted in UML.
Why it matters: Roles make clear who sent each message in a conversation. - Tool result chat continuation: Describe the purpose of continue_chat_with_tool_results() in an existing chat.
Why it matters: Tool results can be returned to the model as part of the conversation. - Prespecified structured output: Describe what structured_output() is designed to produce and how custom classes relate to its generic type.
Why it matters: A specified output format makes model responses easier to fit into an application.
Key terms: BaseLLM, abstract class, async-first, complete(), CompleteResult, prompt, chat(), ChatMessage, ChatRole, chat history
3.1.1 Implementing CompleteResult, ChatMessage, and ChatRole
This section introduces the data structures for completion results and chat messages, including message roles and tool-related messages.
- Completion Result Structure: Describe what CompleteResult stores after a text completion.
Why it matters: This structure keeps a completion’s input and output together. - Chat Message Roles: Explain the purpose of the SYSTEM, USER, ASSISTANT, and TOOL roles in chat messages.
Why it matters: Roles identify how each message contributes to a chat interaction. - Assistant Tool Call Messages: Describe the assistant message carrying a ToolCall in the illustrated chat interaction.
Why it matters: A model response can carry a tool call rather than ordinary message content. - Tool Result Message Conversion: Describe how from_tool_call_result() creates a chat message from a ToolCallResult.
Why it matters: This conversion provides a message carrying a tool result back to the LLM.
Key terms: CompleteResult, prompt, response, ChatMessage, ChatRole, SYSTEM, USER, ASSISTANT, TOOL, ToolCall
3.1.2 Defining BaseLLM
This section defines the BaseLLM interface and explains how it supports tool-result continuation and structured LLM output.
- BaseLLM abstract interface: Describe the BaseLLM interface and the responsibility of its subclasses.
Why it matters: The interface establishes what concrete LLM implementations must provide. - Tool result chat continuation: Trace a tool request through execution and back into the chat using continue_chat_with_tool_results().
Why it matters: This connects tool execution back to an LLM conversation while preserving its history. - Structured output processing rationale: Explain why structured output is easier to use downstream than free-form LLM responses.
Why it matters: Predictable fields make it easier to build code that consumes LLM responses. - Structured output model contract: Describe the inputs and return type of structured_output() and the role of the Pydantic model.
Why it matters: This contract gives callers a typed model for the output they request. - Native tool calling preference: Explain the relationship between tool calling and structured output and why the section favors native tool-calling APIs.
Why it matters: The distinction helps explain why the framework uses native tool calling instead of reconstructing it from structured output.
Key terms: BaseLLM, complete(), chat(), continue_chat_with_tool_results(), structured_output(), ChatMessage, ASSISTANT, ToolCall, ToolCallResult, TOOL
3.2 OllamaLLM: a subclass of BaseLLM
This section introduces OllamaLLM as a BaseLLM subclass that connects the framework to supported models through an Ollama service.
- Ollama integration role: Explain how OllamaLLM integrates Ollama-supported models into the framework.
Why it matters: This integration gives the framework a concrete way to use Ollama-supported models. - OllamaLLM class additions: Describe the roles of the model and _client attributes and the __init__() method in OllamaLLM.
Why it matters: These additions distinguish the Ollama-specific class from the shared BaseLLM blueprint.
Key terms: OllamaLLM, BaseLLM, Ollama service, model, _client, AsyncClient, __init__()
3.2.1 Implementing OllamaLLM
This section implements OllamaLLM configuration and its four BaseLLM interaction methods using Ollama’s client interfaces and data types.
- OllamaLLM instance configuration: Describe how model, host, and think configure an OllamaLLM instance.
Why it matters: Configuration determines which model and Ollama service the integration uses. - Cloud and local setup: Describe the setup choices for running the section’s Ollama notebook examples.
Why it matters: These choices let you run the examples with either a cloud model or a local one. - Completion interface integration: Trace how OllamaLLM implements complete() using Ollama’s completion interface.
Why it matters: This connects the framework’s completion method to Ollama while preserving the framework’s result type. - Structured output integration: Trace how structured_output() requests and returns an instance of the desired Pydantic model.
Why it matters: The schema tells Ollama the desired structure, and the conversion returns that structure as a framework-facing model instance. - Chat message and tool conversion: Trace how chat() converts framework inputs and tools for Ollama and returns framework messages.
Why it matters: These conversions bridge the framework’s chat interface and Ollama’s different message and tool types. - Tool result chat continuation: Trace how continue_chat_with_tool_results() accepts tool results and continues an Ollama chat.
Why it matters: The continuation method lets callers supply tool results directly while the integration handles the message conversions.
Key terms: model, host, think, _client, OLLAMA_MODEL, OLLAMA_API_KEY, ollama_settings(), ollama pull, complete(), ollama.AsyncClient
3.2.2 Hailstone tool call with OllamaLLM
This section demonstrates a complete Hailstone tool-calling process with OllamaLLM, from requesting a tool call to returning its result for an LLM response.
- Hailstone tool calling process: Trace the Hailstone tool-calling process using OllamaLLM, from the initial chat request through tool execution and the continued response.
Why it matters: This sequence connects an LLM's tool request to an actual tool result and a synthesized response.
Key terms: OllamaLLM, hailstone_tool, chat(), ToolCallResult, continue_chat_with_tool_results(), ChatMessage
3.3 Extending to MAS: LLMs
This section explains how backbone LLM choices affect multi-agent systems and how the existing LLM integrations can be reused across agents.
- Backbone LLM selection: Explain how choosing backbone LLMs for different agents affects a multi-agent system’s effectiveness and efficiency, using the coordinator and subordinate example.
Why it matters: Matching model capability and cost to agent roles helps a system handle tasks effectively and efficiently. - LLM reuse across agents: Explain why the framework’s existing LLM work is sufficient for both a single agent and a multi-agent system.
Why it matters: The same LLM foundation can support multiple agents without a separate MAS-specific LLM implementation. - Alternative LLM provider integrations: Describe how OpenAILLM and AnthropicLLM relate to BaseLLM and OllamaLLM and where they can be used.
Why it matters: These integrations offer alternatives to Ollama for the book’s later examples.
Key terms: backbone LLM, coordinator LLM agent, subordinate LLM agents, LLM agent, MAS, OpenAILLM, AnthropicLLM, BaseLLM, OllamaLLM
3.4 Summary
Where this is used later
- (chapter intro) (chapter 4) · The framework’s LLM blueprint.
- LLMAgent: a simple LLM agent class (chapter 4) · The framework's LLM abstraction.
- Implementing LLMAgent: Part 2 (chapter 4) · ChatMessage and chat roles.
- Implementing LLMAgent: Part 1 (chapter 4) · The BaseLLM abstraction used by the agent.
- Designing a processing loop (chapter 4) · The agent processing loop builds on tool calls and LLM responses.
- Implementing a processing loop for LLMAgent (chapter 4) · The agent processing loop builds on the idea of requesting tools, executing them, and returning results to the LLM.
- MAS patterns using subagents (chapter 9) · Multi-agent patterns involve agents with different roles that can benefit from appropriate backbone choices.
- Supporting subagents in our framework (chapter 9) · Support for subagents builds on agents that already use backbone LLMs.
Concepts explained in this chapter
- continue_chat_with_tool_results()continue_chat_with_tool_results() is a BaseLLM method that submits tool results in an ongoing chat interaction for further LLM synthesis and response.
- structured_output()structured_output() is a BaseLLM method that prompts an LLM to return a response conforming to a specified structure or custom type.
- chat()chat() is a framework LLM interface method that processes input messages, conversation history, and optional tools to produce chat messages.
- CompleteResultCompleteResult is a data structure containing an LLM text-completion response and the prompt that produced it.
- ChatMessageChatMessage is a structured representation of a message in an LLM conversation, including its role and content.
- complete()complete() is a BaseLLM method that performs text-completion interaction with a language model.
- BaseLLMBaseLLM is the example framework's abstract class defining an interface for large language models.
- OllamaLLMOllamaLLM is an LLM interface implementation that connects an agent framework to locally running models supported by Ollama.
- tool useToolCall is a data structure representing a requested tool invocation and its arguments.
- ollama.chat()ollama.chat() is an Ollama client method that sends messages and optional tools to a model and returns its chat response.
- ToolCallResultToolCallResult is a framework data structure that records a tool-call identifier, its execution result, and whether an error occurred.
- ChatRoleChatRole is an enumeration of the valid roles for messages in an LLM conversation.
- multi-agent systemA multi-agent system is a system composed of multiple interacting agents that cooperate, coordinate, or compete to achieve goals.
- AsyncClientollama.AsyncClient is Ollama's asynchronous client class for communicating with a running Ollama service.
- BaseModel.model_json_schema()BaseModel.model_json_schema() is a Pydantic method that generates a JSON Schema describing a model.
- BaseModelPydantic BaseModel is a base class for defining typed data models that validate and serialize structured data.
- subagentA subagent is an agent dispatched by another agent to work on a subtask contributing to an overall task.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
