chapter two

2 The brain of AI agents: LLMs

 

This chapter covers

  • Selecting the right LLM for agent development
  • Using LLM APIs with LiteLLM
  • Prompt engineering for agents
  • Experiencing LLM limitations through the GAIA benchmark

Throughout this book, we'll build a Research Agent, a system that can interpret requests like "survey recent work on X" or "extract key findings from these PDFs," gather information from multiple sources, and synthesize findings into comprehensive answers. At the heart of this agent lies the LLM, serving as its decision-making brain. But before we can build an effective agent, we need to answer some fundamental questions: Which LLM should we use? How do we communicate with it programmatically? And crucially, what can an LLM do on its own, and where does it fall short?

We’ll address these questions head-on, starting with exploring how to choose an LLM for agent development, then get hands-on with APIs using LiteLLM to work seamlessly across providers. We'll examine prompt engineering principles that transform a general-purpose LLM into a reliable agent. Finally, we'll put LLMs to the test using the GAIA benchmark, directly experiencing their limitations when faced with real-world problems. This experiment will reveal exactly why agents need tools, setting the stage for everything we build in chapter 3 and beyond.

Figure 2.1 Journey through the book – Chapter 2 in focus

2.1 Choosing LLMs for agents

2.1.1 Starting with closed LLMs

2.1.2 Expanding to open LLMs

2.1.3 Essential LLM capabilities for agents

2.2 LLM API basics for building agents

2.2.1 Setting up the development environment

2.2.2 Getting started with the OpenAI API

2.2.3 Unifying providers with LiteLLM

2.2.4 Conversation management: Handling stateless APIs

2.2.5 Structured output

2.2.6 Asynchronous calls

2.3 Enhancing agent intelligence: Prompt engineering

2.3.1 The role of system prompts

2.3.2 Guidelines for agent prompts

2.4 Experiencing LLM limitations with GAIA

2.4.1 Why GAIA? Setting goals for agent development

2.4.2 Experiment setup

2.4.3 Results and analysis: Why LLMs need tools

2.5 Summary