Chapter 2: Core components: Large language models, prompting, and agents

Introduction

The chapter introduction identifies the roles of the LLM, prompting, and OpenAI Agents SDK in an agent’s core architecture.

• Core agent component roles: Explain that the large language model serves as an agent’s brain, prompting shapes its reasoning, and the OpenAI Agents SDK orchestrates these components.
  Why it matters: These roles provide a starting point for understanding the components built later in the chapter.

Key terms: large language model, prompting techniques, OpenAI Agents SDK, brain

2.1 Understanding large language models

This introduction presents LLMs as widespread tools for AI agents and notes that they can be useful without a complete understanding of their internal intelligence.

• LLMs in AI agents: Explain that LLMs are widespread in AI and that agents powered by them demonstrate the potential of combining the two.
  Why it matters: This is why the book treats LLMs as important to building agents.
• Useful despite incomplete understanding: Explain why treating LLMs as black boxes does not prevent people from using them.
  Why it matters: A complete account of an LLM's internals is not a prerequisite for using it.

Key terms: large language models (LLMs), AI agents, black boxes, internal intelligence

2.1.1 LLMs: Probabilistic token machines

The section explains how LLMs learn to predict tokens and generate text, and distinguishes base training from alignment.

• Next-token prediction and sampling: Trace how text becomes numbered tokens, how the model assigns probabilities to possible next tokens, and how a separate decoding step selects an output token. Explain that generation repeats until an EOS token stops it.
  Why it matters: This is the basic process behind the text an LLM produces.
• Training through prediction error: Explain how base training compares a predicted next token with the expected token, calculates error, and uses backpropagation to adjust the model’s weights over many examples.
  Why it matters: It explains what the model changes while learning from training text.
• Base training versus alignment: Distinguish base training from the later alignment phase: explain that RLHF aims to improve instruction-following, helpfulness, and avoidance of harmful output, rather than teaching new knowledge or improving reasoning.
  Why it matters: It prevents confusing a model’s conversational behavior with gains in knowledge or reasoning.
• Learned weights during inference: Explain that the model’s predictive capability resides in weights learned by its attention and feedforward layers, not in a lookup table or stored probability map. Describe how each forward pass uses those weights and the current context to produce a next-token distribution.
  Why it matters: It clarifies why fluent output is not evidence that the model is looking up a stored answer.

Key terms: tokenization, tokens, vocabulary, probability distribution, sampling, decoding strategy, base training, expected token, loss, backpropagation

2.1.2 What is a token?

This section explains how LLMs turn text into tokens, why token counts differ from text length, and why those counts matter for cost and output.

• Tokens and vocabulary indices: Explain how tokenization breaks text into words or word parts, maps them to numerical vocabulary indices, and lets an LLM predict the next token.
  Why it matters: Tokens are the units the model consumes and predicts.
• Text length versus token count: Explain why raw text length is not a reliable guide to token count, using the section's JSON-versus-regular-text comparison.
  Why it matters: A compact-looking representation may still use more of an LLM's tokens.
• Input and output token costs: Describe how providers price input and output tokens separately and why token use can become costly in large agentic systems.
  Why it matters: Token counts have a direct effect on the cost of running an agent.
• Model-specific token measurement: Describe how tiktoken can measure tokens in supplied text using a model's vocabulary encoding, and note that the OpenAI Agents SDK reports token counts per interaction.
  Why it matters: Measuring tokens gives a better count than guessing from visible text length.
• Input volume and uncertainty: State the section's caution about how a larger number of input tokens typically affects LLM output.
  Why it matters: Input size is relevant to output behavior as well as cost.

Key terms: tokenization, tokens, numerical indices, vocabulary, token count, JSON, regular text, overhead, input tokens, output tokens

2.1.3 Tuning temperature, top-p, and more

This section explains how generation parameters affect LLM output, how to tune them for an agent’s role, and why their effects require care.

• Sampling and output controls: Distinguish parameters that affect token sampling from max tokens, which caps response length, and from API settings that control the call itself.
  Why it matters: This distinction helps you choose a setting that addresses the output behavior you actually want to change.
• Role based parameter choices: Explain why a coder agent and a creative writer agent might use different generation settings.
  Why it matters: The desired kind of output determines which trade-offs are worth making.
• Limits of generation tuning: Explain why generation settings should not be treated as guarantees and why temperature and top-p are generally not adjusted together.
  Why it matters: Treating tuning as an influence rather than a guarantee prevents false confidence in an agent’s output.
• Practical length and seed settings: Describe the usual parameters to adjust, the purpose of a max-tokens cap, and how a seed helps with repeatable runs.
  Why it matters: These settings offer a practical starting point for controlling response length and repeating a run when investigating results.

Key terms: sampling, temperature, top-p, presence penalty, frequency penalty, max tokens, coder agent, creative writer agent, seed

2.2 Controlling LLMs with prompt engineering (agent persona)

This section explains why clear, well-structured prompts remain central to making agents reliable, even as LLMs improve.

• Prompt engineering and agent reliability: Explain why prompt engineering still matters for agents despite improvements in LLMs, and how it relates to the agent’s persona.
  Why it matters: A good prompt helps an agent act reliably rather than merely produce a plausible response.

Key terms: prompt engineering, persona, clear, well-structured prompts

2.2.1 Applying core prompt techniques

The section explains why transferable prompt techniques, clear structure, and cache-aware ordering make agent instructions more effective.

• Transferable prompt techniques: Explain why some prompt techniques work across LLMs even though wording and tone may need to vary, and describe how those techniques can inform an agent’s persona or base instructions.
  Why it matters: It helps you distinguish reusable prompting principles from details your agent framework already handles.
• Clear prompt structure: Describe how Markdown and unique delimiters organize a research agent prompt, and explain why the section favors clear structure despite the added tokens.
  Why it matters: Clear sections make agent instructions easier for an LLM to follow consistently.
• Cache-aware prompt ordering: Explain how placing stable prompt content before dynamic content increases prompt-cache reuse in high-traffic systems.
  Why it matters: Prompt layout can affect production cost and latency, not just response clarity.

Key terms: prompt techniques, agent persona, base set of instructions, OpenAI Agents SDK, Markdown, unique delimiters, structure, loss feedback, prompt caching, static portion

2.2.2 Thinking like an LLM

This section explains how to write prompts as clear task instructions, express workflows in those instructions, and recognize a limit of prompting an LLM to repeat tasks.

• Specific task instructions: Explain why a prompt should give specific instructions without assuming the LLM has background knowledge of the task.
  Why it matters: Specific instructions give the LLM a clearer target for its response.
• Workflow instructions in prompts: Describe how a prompt can tell an agent how to carry out a task, including considerations, decisions, examples, and a final output task. Explain the section’s suggested way to check such instructions.
  Why it matters: A workflow prompt makes the intended steps and decisions explicit enough to review before using it.
• Limits of prompted task loops: Distinguish what the section says about an LLM following a task loop written in a prompt from an agent continuing that loop.
  Why it matters: Writing a loop in a prompt does not by itself mean the LLM will keep trying until the goal is resolved.

Key terms: prompt, instructions, task, workflow, prompt instructions, decision-making, consolidated output task, task loops, LLM, agent

2.2.3 Avoiding common prompt pitfalls

This section cautions against over-engineering agent prompts and recommends splitting prompts that become too large or complicated.

• Avoiding over-engineered agent prompts: Explain why an LLM-written or revised prompt still needs to be kept from becoming overly complicated or explicit, and what to do when a prompt grows too large.
  Why it matters: A prompt that keeps growing may be better handled as smaller tasks rather than more instructions.

Key terms: over-engineering, prompt instructions, smaller tasks

2.3 Building an agent with OpenAI Agents

The section places the OpenAI Agents SDK among varied agent frameworks and explains the appeal of shared protocols and a flexible SDK.

• Shared Agent Interaction Protocols: Explain that agent frameworks can implement agents differently, while protocols such as MCP and A2A provide a common foundation for interactions with tools and other agents.
  Why it matters: It separates a framework’s approach to building an agent from the protocols used for agent interactions.
• OpenAI Agents SDK Rationale: Explain why the section describes the OpenAI Agents SDK as light, extensible, and flexible: it arrived after earlier agent frameworks and benefited from their lessons.
  Why it matters: This is the section’s stated reason for favoring the SDK used in the book.

Key terms: frameworks, MCP, A2A, protocols, OpenAI Agents SDK, light, extensible, flexible

2.3.1 Building a minimal agent

This section introduces a minimal research planning agent and points out that its five-step output follows its prompt instructions.

• Research planning agent output: Describe what the example agent produces and how its output relates to its prompt.
  Why it matters: The example shows prompt instructions shaping a simple agent’s response.

Key terms: research planning agent, five-step plan, prompt instructions

2.3.2 Setting the agent model and other parameters

This section explains how model selection, temperature, and reasoning effort affect an agent’s consistency, cost, and speed.

• Temperature and output consistency: Explain why this agent uses temperature 0 for planning and why repeated runs can still vary.
  Why it matters: Consistent plans are easier to debug, but you should not mistake reduced variation for a guarantee.
• Explicit agent model selection: Explain why the example specifies an exact model while leaving other model parameters at their defaults.
  Why it matters: Naming the model makes the example’s model choice explicit rather than dependent on a provider default.
• Reasoning effort by task: Explain why routine agent steps should generally use none or minimal reasoning effort and when higher effort is warranted.
  Why it matters: Selective deliberation keeps routine agent work fast and inexpensive without ruling out deeper thought when needed.

Key terms: temperature, highest-probability token, output variation, exact model, SDK, defaults, reasoning-effort parameter, thinking, reasoning_effort, latency

2.3.3 Controlling inputs and typed outputs

This section shows how typed agent outputs make workflows more reliable, why a proposed task-output type fails strict JSON validation, and how typing simplifies prompts.

• Typed outputs in agent workflows: Explain why strongly typed inputs and outputs help one agent pass results to another, and describe how the planning agent uses a Pydantic data model to request a list of tasks.
  Why it matters: A predictable output structure helps keep a chain of agents from breaking when an LLM's wording varies.
• Strict JSON schema failure: Explain why the richer task-output type triggers a strict JSON schema error and distinguish the preferred fix from disabling the strict check.
  Why it matters: Fixing the type preserves the schema check that helps catch troublesome output structures.
• Simpler prompts through typing: Describe how typed outputs reduce the need to prescribe an exact output format in the prompt.
  Why it matters: Typing can make prompts simpler as well as make agent outputs easier to pass along.

Key terms: typed outputs, inputs, Pydantic, data model, list of tasks, strict JSON schema, dict[int, str], extra properties, strict_json_schema=False, typing outputs

2.3.4 Tracing agents

Explains how to inspect OpenAI agent traces, name a workflow trace, and choose an external tracing layer when appropriate.

• Inspecting agent execution traces: Explain when traces appear on the OpenAI Traces page and describe how to inspect a workflow’s response output.
  Why it matters: Inspecting traces helps you see what an agent did, especially as its workflows grow more complex.
• Named workflow traces: Describe what setting a trace in code adds to the default tracing view.
  Why it matters: A named workflow makes the relevant run easier to select and inspect.
• Choosing an external tracing layer: Distinguish the built-in OpenAI Traces dashboard from external tracing options and explain when the latter may be preferable.
  Why it matters: The right tracing layer depends on whether an agent must be observed beyond a single provider’s dashboard.

Key terms: tracing, OpenAI Traces, workflow, ResearchPlanModel, trace, framework-agnostic tracing, external tracing layer

2.4 Enhancing agents through tool integration

This section distinguishes a prompt-chaining workflow from an agent with agency and identifies tool choice as a way to give agents that agency.

• Agency versus prompt chaining: Distinguish an agent with agency from a workflow made of prompt steps, using the research planner as the example.
  Why it matters: It helps you distinguish a useful prompt-based workflow from an agent that can decide what to do.
• Tool choice and agency: Explain why giving agents tools they can choose when and where to use supports flexible, adaptable behavior.
  Why it matters: The point of tools here is to give the agent a choice of actions, not merely to add features.

Key terms: agency, research planner, prompt step, prompt chaining, tools, flexible and adaptable agents

2.4.1 Providing agents with tools

This section shows how to give an agent tools, guide their use, and limit the costs and risks they introduce.

• Tools and agent actions: Distinguish a tool, which is a capability available to an agent, from an action, which is what the agent does with it. Describe the kinds and locations of tools identified here.
  Why it matters: Separating capabilities from actions makes it easier to reason about what an agent can do and what it actually does.
• Research source tool instructions: Explain why the research planning agent’s prompt explicitly tells it when to use get_research_sources and what it does with the returned sources.
  Why it matters: A tool is more useful when the agent has clear instructions for when and how it fits into its task.
• Internal function tool registration: Describe what @function_tool does when exposing an internal function and what registration means for subsequent LLM calls.
  Why it matters: Registration makes a function available to the agent, but also affects every LLM call.
• Tool count and reliability costs: Explain why adding more tools is not automatically better, including effects on selection, token cost, prompt complexity, and failures. Describe the failure-handling patterns the section names.
  Why it matters: Tool choice affects both the agent’s ability to pick correctly and the cost and reliability of each task.
• Limits on agent authority: Justify limiting an agent’s tools to those it needs by explaining how tool access gives it consequential choices.
  Why it matters: Restricting available tools limits the harm an agent can cause through a mistaken decision.

Key terms: tools, actions, MCP, handoff, get_research_sources, research planning agent, prompt instructions, research sources, @function_tool, JSON tool description

2.4.2 Tracing agentic tool use

This section explains dependent and parallel tool calls and shows how OpenAI Traces makes an agent’s tool use visible for debugging.

• Dependent tool chaining: Explain how one tool’s output can become another tool’s input, using the Planning Agent’s research-source and URL lookups as an example.
  Why it matters: It explains why an agent may need several ordered tool calls to complete one task.
• Parallel versus dependent calls: Distinguish parallel tool calls from sequential chains and explain when parallel calls can reduce latency.
  Why it matters: The dependency between calls determines whether an agent can save time by making them together.
• Inspecting agent tool traces: Describe what the OpenAI Traces page records and how to use that view to inspect tool chains and troubleshoot agent behavior.
  Why it matters: Seeing what the agent called, in what order, and with which values helps make complicated behavior understandable.

Key terms: tool chaining, get_resource_url, parallel, sequential chains, latency, OpenAI Traces page, tool chains, inputs and outputs

2.5 Exercises

Five exercises guide the reader through building a Research Planner agent, comparing temperature settings, enforcing typed output, refining its prompt, and tracing a tool call.

2.6 Summary

This chapter summary connects token-aware model tuning, prompt and schema design, and observable tool use as foundations for reliable agents.