Chapter 2: The brain of AI agents: LLMs

Introduction

Introduces the research agent built throughout the book and identifies the LLM as its decision-making brain.

• Research Agent Decision-Making Brain: Explain what the book’s research agent is meant to do and the LLM’s role within it.
  Why it matters: This gives you a concrete agent to keep in mind as the book develops its components.

Key terms: research agent, large language model (LLM), decision-making brain

2.1 Choosing LLMs for agents

This section frames LLM selection as an early agent-development decision that affects performance, cost, and development speed.

• Impact of LLM choice: Explain why choosing an LLM matters when starting agent development, naming the effects on agent performance, cost, and development speed.
  Why it matters: LLM selection is an early decision with practical consequences for the agent you build.

Key terms: LLM, agent’s performance, cost, development speed

2.1.1 Starting with closed LLMs

Explains the difference between open and closed LLMs and why closed LLMs are a practical starting point for agent development.

• Open and closed LLM distinction: Distinguish open LLMs, whose weights can be downloaded, run locally, and fine-tuned, from closed LLMs, which are accessed through APIs.
  Why it matters: This distinction makes the section’s reasons for starting with closed LLMs easier to follow.
• Rapid agent experimentation: Explain why closed LLMs let developers validate agent ideas and iterate quickly, including the benefit of using the same environment from prototype to production.
  Why it matters: Less model setup leaves more time to test and improve the agent itself.
• Model intelligence and agent performance: Explain why the section favors closed LLMs for complex agent tasks, relating model intelligence to understanding intent, selecting tools, and multistep reasoning.
  Why it matters: The model choice affects whether an agent can handle the decisions its tasks demand.
• Ready-made agent API features: Describe the agent-relevant features closed LLM providers offer through stable APIs and contrast their immediate availability with the extra work open LLMs may require.
  Why it matters: Built-in API features reduce the integration work needed to build an agent.

Key terms: open LLMs, closed LLMs, weights, APIs, rapid experimentation and validation, API key, agent logic, prototype, production, LLM intelligence

2.1.2 Expanding to open LLMs

This section explains why and how to consider open LLMs after validating an agent with closed LLMs.

• Cost and task allocation: Explain when self-hosted open LLMs may reduce costs and how a hybrid setup divides work between models.
  Why it matters: You can consider cheaper models for suitable tasks without assuming they must replace the agent’s core reasoning model.
• Infrastructure stability and control: Explain how self-hosted open LLMs address dependence on external APIs and sensitive-data requirements.
  Why it matters: Model choice can be driven by operational and regulatory constraints, not just model performance or price.
• Gradual open-model transition: Describe the recommended sequence for adopting open LLMs and how to judge a ported design.
  Why it matters: A gradual transition lets you evaluate open models against an already validated agent design rather than changing everything at once.

Key terms: self-hosting, classification, extraction, core reasoning, auxiliary tasks, external APIs, self-hosted open LLMs, sensitive data, closed LLMs, open LLMs

2.1.3 Essential LLM capabilities for agents

Identifies LLM capabilities agents need and explains why flexible model selection matters as models change.

• Tool calling for external actions: Explain why an agent needs an LLM that can select and invoke tools to interact with the external world.
  Why it matters: Tool calling is what allows an agent to do more than produce text.
• Structured output for integration: Explain why agents need LLM output that follows JSON or another specified schema.
  Why it matters: Software integrations need responses in a form they can use.
• Context capacity and relevance: Explain why agents need sufficient context capacity while distinguishing capacity from effective context selection.
  Why it matters: A large context window helps only when the agent receives useful information.
• Model agility and provider choice: Justify designing agent code so models and providers can be switched, and explain how that fits the book’s choice of OpenAI.
  Why it matters: Flexible model access keeps an agent adaptable without making every example depend permanently on one provider.
• Reasoning depth and tradeoffs: Distinguish reasoning models’ potential benefit and cost, and describe the section’s point about models that adapt reasoning depth.
  Why it matters: Reasoning depth is a model-selection tradeoff, not a free improvement for every task.

Key terms: tool calling, function calling, structured output, JSON, schemas, long context support, context window, context engineering, model agility, LiteLLM

2.2 LLM API basics for building agents

This section introduces the upcoming hands-on work with LLM APIs and the capabilities needed for agent development.

2.2.1 Setting up the development environment

This section sets up the companion notebooks and explains how they obtain the API keys needed for LLM calls.

• Companion notebook environment: Describe how to prepare the companion code: clone the GitHub repository, use uv with its pyproject.toml to create a virtual environment and install dependencies, then launch Jupyter Lab.
  Why it matters: This setup lets you run the book’s examples in their intended environment.
• API key loading from notebooks: Explain where to put the required credentials and how notebooks load them before making LLM calls. Distinguish the OpenAI API key needed for the examples from the HF_TOKEN needed for the GAIA dataset.
  Why it matters: Loading the root .env file makes credentials available to notebooks even when they run from chapter subdirectories.

Key terms: uv, pyproject.toml, virtual environment, Jupyter Lab, .env, OpenAI API key, HF_TOKEN, python-dotenv, find_dotenv(), environment variables

2.2.2 Getting started with the OpenAI API

This section explains the basic inputs and outputs of an OpenAI Chat Completions call, common generation controls, and how the API compares with another provider’s.

• Model and conversation inputs: Explain what the model and messages parameters supply to chat.completions.create.
  Why it matters: These are the two main inputs to the API call used in this section.
• Message objects and roles: Describe the structure of messages and distinguish the developer, user, and assistant roles.
  Why it matters: Roles tell the API how each piece of conversation content should be interpreted.
• Response text location: State where to find generated text in an OpenAI response and why choices is a list.
  Why it matters: An API call is useful only if you can extract the model’s reply from its response.
• Generation control parameters: Explain how temperature affects output and what max_tokens limits.
  Why it matters: These parameters let you match response consistency and length to the task.
• Provider interface similarities: Distinguish the broad similarity of LLM provider APIs from the specific differences between OpenAI and Anthropic shown here.
  Why it matters: Similar APIs transfer familiar ideas across providers, but their differences still require attention.

Key terms: chat.completions.create, model, messages, role, content, developer, user, assistant, response.choices[0].message.content, choices

2.2.3 Unifying providers with LiteLLM

This section explains how LiteLLM simplifies comparing and switching LLM providers through one interface.

• LiteLLM provider unification: Explain why LiteLLM is useful when developing agents and describe how its unified interface handles calls, model selection, responses, and API keys.
  Why it matters: A common interface makes it easier to compare models and switch providers for cost or performance.

Key terms: LiteLLM, unified interface, completion function, OpenAI format, environment variables

2.2.4 Conversation management: Handling stateless APIs

Explains why LLM API conversations require the caller to resend history, and why growing agent context has a cost.

• Stateless LLM API calls: Explain that each LLM API call is independent: unlike the apparent memory of a chat web service, a later call does not remember information supplied in an earlier one.
  Why it matters: This explains why conversation continuity must be supplied by the application rather than assumed from the API.
• Manual conversation history: Describe how to maintain conversation continuity by accumulating messages and sending the entire history with each call, marking user messages and model responses with their respective roles.
  Why it matters: Resending the history lets the model respond coherently to earlier turns.
• Agent context growth costs: Explain why agents need to include more than conversation turns in their context and what happens to token use as that context grows.
  Why it matters: Agent context can become expensive and slow as it accumulates more than ordinary chat turns.

Key terms: stateless, API call, messages list, conversation history, user role, assistant role, context, tool-call results, search results, intermediate reasoning steps

2.2.5 Generating structured output

This section explains why agents need structured LLM responses and how to request a validated output format with Pydantic.

• Purpose of structured output: Explain why an agent needs structured output rather than free-form text when choosing actions, including a tool and its arguments.
  Why it matters: A defined response format lets agent code turn an LLM’s decision into an action it can process.
• Pydantic output schema: Describe how a Pydantic model defines and validates the expected output, and how passing it to response_format produces a parsed result.
  Why it matters: A schema makes the response’s expected shape explicit and helps catch malformed data early.

Key terms: structured output, tool calling, arguments, JSON, Pydantic, BaseModel, schema, response_format, content attribute, ExtractedInfo

2.2.6 Asynchronous calls

This section shows how to run LLM requests concurrently while limiting API calls and handling failures.

• Concurrent LLM request execution: Explain why asynchronous LLM calls can reduce elapsed time, and describe how async def, await, LiteLLM’s acompletion, and asyncio.gather work together in the section’s example.
  Why it matters: Overlapping requests makes batches such as benchmark evaluations faster than waiting for each call in turn.
• Concurrency limits and retries: Describe how the section limits simultaneous API calls with asyncio.Semaphore and handles transient failures with LiteLLM’s num_retries.
  Why it matters: Concurrency is useful only if a batch does not overwhelm the API and can recover from temporary failures.
• Independent batch failure handling: Explain what return_exceptions=True changes in asyncio.gather when one request in a batch fails.
  Why it matters: A single failed benchmark problem need not discard results from the rest of the batch.

Key terms: async def, coroutines, await, asyncio.gather, acompletion, asyncio.Semaphore, async with, num_retries, exponential backoff, return_exceptions=True

2.3 Enhancing agent intelligence: Prompt engineering

The section distinguishes user prompts from system prompts and explains why developer-defined system prompts help an LLM act consistently as an agent.

• User and system prompt roles: Distinguish user prompts, which users supply and change each turn, from system prompts, which developers define and keep in place throughout a conversation.
  Why it matters: The distinction shows which instructions a developer can use to shape an agent across different user requests.
• System prompts for agent behavior: Explain how a system prompt guides a general-purpose LLM toward reliable behavior on specific tasks despite varied user inputs.
  Why it matters: A system prompt gives developers a way to guide agent behavior even when they cannot control what users ask.

Key terms: user prompts, system prompts, system prompt, personality, permissions, constraints, tool-use policies

2.3.1 The role of system prompts

This section explains how system prompts guide an agent’s decisions by defining its identity, response style, behavioral boundaries, and knowledge limits.

• Persistent behavioral guidance: Explain why system prompts matter especially for agents: they remain in the context and guide decisions across multiple autonomous steps.
  Why it matters: The prompt influences not just one reply but the agent’s decisions throughout a task.
• Agent identity and capabilities: Describe how a system prompt defines an agent’s role and capabilities so it can accurately tell users what it can do and when to refer them elsewhere.
  Why it matters: Users need an accurate account of what the agent can handle.
• Situation-appropriate response style: Explain how system-prompt instructions about tone and formatting can counter an LLM’s tendency to use excessive lists and produce more appropriate responses.
  Why it matters: Response style affects whether an agent’s answers feel natural and useful.
• Limits on autonomous behavior: Explain why a system prompt must state what an autonomous agent must not do, including limits on its authority.
  Why it matters: Clear boundaries keep an agent from exceeding the authority it was meant to have.
• Honest knowledge limits: Describe how a system prompt can tell an agent to acknowledge knowledge limits and use an appropriate tool or tell the user when a request exceeds them.
  Why it matters: Acknowledging limits and offering alternatives is more trustworthy than inventing an answer.

Key terms: system prompt, context, behavioral rules, product identity, role, capabilities, output format, style, tone, bullet points

2.3.2 Guidelines for agent prompts

Explains how agent prompts should encourage autonomous action, guide when to use or avoid tools, and provide clear tool information.

• Autonomous action and clarification: Explain why an agent should act without asking permission when the user’s intent is clear, but ask minimal clarifying questions when it is genuinely ambiguous.
  Why it matters: This distinction helps the agent make progress without repeatedly handing straightforward decisions back to the user.
• Proactive tool use triggers: Describe how prompt guidelines can specify when to use tools proactively, including current-information needs and phrases that signal a need to search past conversations.
  Why it matters: Concrete triggers give the agent a better basis for deciding when a tool will help.
• Negative tool use guidance: Explain when a prompt should tell an agent not to search and why unnecessary tool calls are undesirable.
  Why it matters: Avoiding needless searches keeps straightforward answers fast and the agent less costly to use.
• Concrete tool behavior examples: Describe how example user requests, actions, and responses can anchor a prompt’s tool-use guidance.
  Why it matters: Examples show the agent what the desired decision looks like in realistic requests.
• Tool descriptions versus guidelines: Distinguish system-prompt guidelines about when to use tools from descriptions of what each tool does, requires, and returns.
  Why it matters: Good timing instructions cannot compensate for an unclear account of what a tool can do.

Key terms: agent, chatbot, intent, system prompt, clarification, tool-use guidelines, default attitude, trigger patterns, linguistic signals, conversation search tool

2.4 Experiencing LLM limitations with GAIA

This section introduces an upcoming experiment using real problems to observe LLM capabilities and limitations and motivate the need for agent tools.

2.4.1 Why GAIA: Setting goals for agent development

This section explains why GAIA provides a measurable, user-relevant target for developing a research agent and introduces its difficulty levels and problem fields.

• Measurable agent development goals: Explain why agent development needs a concrete benchmark and what makes test cases useful for measuring a research agent’s progress.
  Why it matters: A benchmark makes it possible to judge improvements instead of relying on impressions.
• GAIA as user requests: Describe how to interpret GAIA problems as simulated requests to a research agent and distinguish the benchmark score from the broader development goal.
  Why it matters: Treating benchmark questions as user requests keeps development focused on useful behavior, not just a score.
• GAIA difficulty and baseline: Distinguish the three GAIA difficulty levels at the depth described here and explain why the chapter starts with level 1.
  Why it matters: Starting with easier problems gives the later tool-equipped agent a baseline for comparison.
• GAIA fields and tool clues: Describe what the named GAIA problem fields mean and what the Kipchoge problem’s annotator metadata reveals.
  Why it matters: The fields tell you what to ask, how to check the answer, and what resources solving the question may demand.

Key terms: benchmark, test cases, verifiable answers, difficulty levels, GAIA, simulated user inputs, verified answers, level-1, level-2, level-3

2.4.2 Experiment setup

This section sets up a GAIA experiment that collects comparable LLM answers, manages API calls, checks correctness, and records results across models.

• Comparable GAIA response format: Explain how structured output and GAIA’s answer-formatting instructions make model responses comparable with ground truth.
  Why it matters: A consistent response makes it possible to compare answers and distinguish inability to answer from an attempted answer.
• Provider-limited problem solving calls: Trace how a single problem is sent to a model while respecting provider-specific concurrency limits and handling transient failures or refusals.
  Why it matters: This lets the experiment make many calls without treating provider limits or model refusals as ordinary answers.
• Exact-match answer validation: Describe how a predicted answer is checked against GAIA’s ground truth and what happens when no prediction is available.
  Why it matters: The comparison rule determines which outputs count as correct in the experiment.
• Per-problem evaluation metadata: Explain what evaluate_gaia_single records and why those records help analyze failures.
  Why it matters: A correctness score alone cannot show what kind of failure occurred.
• Concurrent multi-model experiment: Describe how the experiment runs problem–model combinations and why it begins with a small dataset subset.
  Why it matters: Running combinations concurrently makes a multi-model evaluation practical, while a small initial run checks the setup before greater expense.

Key terms: structured output, is_solvable, unsolvable_reason, final_answer, ground truth, semaphore, solve_problem, async with, acompletion, num_retries

2.4.3 Results and analysis: Why LLMs need tools

The GAIA results show why tool-free LLM performance is limited, why apparent successes can be unreliable, and what external capabilities agents need.

• Tool-free GAIA baseline: Explain why only five of the 20 level-1 GAIA problems were considered reliably solvable without tools, and what the 25% figure means for interpreting the experiment.
  Why it matters: This baseline makes it possible to distinguish reliable tool-free capability from occasional correct answers.
• Model self-assessment differences: Describe how the models differed in deciding which problems to attempt and what good calibration meant in this experiment.
  Why it matters: An agent can make better use of tools when its model recognizes that an answer needs outside information.
• Training knowledge versus sources: Explain why models sometimes answered apparently tool-dependent questions correctly, and why those successes do not make training data a dependable substitute for tools.
  Why it matters: A correct-looking answer from remembered training material is not a reliable replacement for checking a source.
• Missing external capabilities: Describe why web-dependent and file-dependent GAIA problems cannot be reliably solved from the model alone, using the section's examples.
  Why it matters: The missing capability depends on where the needed information is and what the user asks the agent to verify.
• Model refusals in production: Describe how an apparently harmless GAIA question triggered refusals in some models and what the section recommends for handling refusals in production agents.
  Why it matters: An agent must cope with refusals that vary across models, even when the user's underlying request is simple.

Key terms: level-1 GAIA problems, theoretical maximum accuracy, without tools, Judged solvable, calibration, external information, training data, confabulating, authoritative sources, web search

2.5 Summary

This summary brings together the chapter’s practical guidance on choosing and using LLMs for agents, prompting them, and recognizing when they need tools.