Chapter 13: Creating end-to-end LLM applications
The chapter introduction contrasts standalone SLM use with larger language-model systems and states why those systems will use domain-specific small models.
After this chapter you will be able to
- Standalone and system-based use. Describe the standalone pattern in which an SLM responds to a prompt and returns a result, and explain why the chapter considers its use within a larger system.
- Whole-system business value. Explain why an LLM application must be designed as an integrated system around a business goal, using the Formula 1 engine-and-car analogy.
- Model fit and compliance. Justify choosing a smaller, specialized model that fits the application instead of optimizing for the largest possible model, including the section’s point about regulatory compliance.
- Offline and online RAG phases. Trace the offline preparation of a document corpus and the online path from a user query to a generated answer.
- Shared-space semantic retrieval. Explain how the example retrieves relevant chunks using embeddings, including why the query uses the document embedding model.
- Retrieved context for generation. Describe how the example turns retrieved chunks and the user’s question into input for the local generator model.
- Vector database retrieval. Explain when a RAG application benefits from a vector database and how it retrieves records using a query vector.
- LanceDB ingestion and search. Trace how the existing embeddings are prepared, stored in LanceDB, indexed, and searched in the example.
- Agentic capabilities and purpose. Distinguish an AI agent from a prompt-in, output-out application. Explain how an agent’s reasoning, actions, and interactions help it pursue a user’s goal despite limitations of a language model alone.
- CodeAgent reasoning and execution. Trace how smolagents’ CodeAgent uses a model to reason about a task, choose tools, generate Python actions, and work toward an answer. Describe the execution limits and safeguards stated for this implementation.
- Agent components and tool setup. Describe the components and configuration used to create the section’s CodeAgent, including its model, toolbox, custom tools, and run method.
- Flight booking tool workflow. Trace the local flight-booking example from a user task through flight search, seat selection, and mock booking. Explain what the unrelated Fibonacci tool demonstrates.
What it builds on
- Domain-specific vs. generalist LLMs for business value (chapter 1) · Introduces the business-value comparison between domain-specific and generalist models.
- Retrieval-augmented generation (chapter 2) · The earlier introduction to retrieval-augmented generation.
- Offline serving (chapter 11) · The Phi-3 Mini GGUF model previously used for offline serving.
- Running an LLM locally with Ollama (chapter 12) · Running an LLM locally, an option this section also uses for agent model serving.
Section by section
Introduction
The chapter introduction contrasts standalone SLM use with larger language-model systems and states why those systems will use domain-specific small models.
- Standalone and system-based use: Describe the standalone pattern in which an SLM responds to a prompt and returns a result, and explain why the chapter considers its use within a larger system.
Why it matters: This distinction explains the chapter’s shift from running a model to building an application around one. - Domain-specific small model benefits: State the benefits the introduction attributes to using domain-specific small models as the basis for the chapter’s larger systems.
Why it matters: The model choice is part of the chapter’s reason for considering these application designs.
Key terms: SLM, larger system, domain-specific small models, privacy, safety, security, specificity, cost savings
13.1 Why LLMs alone aren’t enough
This section argues that business value comes from a complete, well-integrated LLM application rather than from making the language model as large as possible.
- Whole-system business value: Explain why an LLM application must be designed as an integrated system around a business goal, using the Formula 1 engine-and-car analogy.
Why it matters: It keeps the application’s business goal—not model size—at the center of design decisions. - Model fit and compliance: Justify choosing a smaller, specialized model that fits the application instead of optimizing for the largest possible model, including the section’s point about regulatory compliance.
Why it matters: Model choice affects both integration with the application and the ability to meet real-world constraints. - Limits of model-only improvement: Distinguish the useful routine tasks of large models from the section’s argument that scaling models or relying only on prompt engineering is insufficient for broader application goals.
Why it matters: It explains why the section looks beyond both model scaling and prompts when considering practical applications.
Key terms: language model, final system, business goal, smaller models, tailored SLM, full stack, compliance, AGI, prompt engineering, external knowledge
13.2 Combining a domain-specific SLM with RAG
The section builds a local RAG example that prepares a PDF for retrieval, finds relevant passages with a small embedding model, and supplies those passages to a small generator model.
- Offline and online RAG phases: Trace the offline preparation of a document corpus and the online path from a user query to a generated answer.
Why it matters: The two phases explain what must be prepared in advance and what happens for each question. - PDF document chunk preparation: Describe how the example turns PDF pages into text chunks suitable for embedding and why it splits the text.
Why it matters: Retrieval depends on having manageable pieces of source text to embed and return. - Shared-space semantic retrieval: Explain how the example retrieves relevant chunks using embeddings, including why the query uses the document embedding model.
Why it matters: A shared vector space lets the system retrieve relevant context even when a passage uses different wording from the question. - Retrieved context for generation: Describe how the example turns retrieved chunks and the user’s question into input for the local generator model.
Why it matters: Finding relevant passages only helps the answer if those passages are supplied to the generator.
Key terms: RAG, offline, online, embeddings, vector database, generator model, PDF, sentences, chunks, token limit
13.3 Using a vector database
This section replaces disk-persisted embeddings in a small RAG example with a LanceDB vector database and compares the resulting retrieval and answer quality.
- Vector database retrieval: Explain when a RAG application benefits from a vector database and how it retrieves records using a query vector.
Why it matters: It explains why the section changes the retrieval system rather than just the language model. - LanceDB ingestion and search: Trace how the existing embeddings are prepared, stored in LanceDB, indexed, and searched in the example.
Why it matters: It connects the earlier RAG data to the indexed retrieval used in this example. - Retrieval quality comparison: Distinguish the example's observed results from the larger-scale benefit claimed for indexed retrieval.
Why it matters: A higher retrieval score in this small example should not be mistaken for a demonstrated improvement in its generated answer.
Key terms: vector database, vector store, embeddings, approximate nearest neighbor (ANN), query vector, LanceDB, NumPy arrays, list of dictionaries, PyArrow table, custom schema
13.4 Building an agent
The section explains what makes a system agentic and builds a local flight-booking agent that plans steps and uses custom tools.
- Agentic capabilities and purpose: Distinguish an AI agent from a prompt-in, output-out application. Explain how an agent’s reasoning, actions, and interactions help it pursue a user’s goal despite limitations of a language model alone.
Why it matters: This distinction helps you judge whether an application actually delegates decisions and actions rather than merely using model-generated text. - CodeAgent reasoning and execution: Trace how smolagents’ CodeAgent uses a model to reason about a task, choose tools, generate Python actions, and work toward an answer. Describe the execution limits and safeguards stated for this implementation.
Why it matters: CodeAgent’s generated code is the link between a model’s plan and the actions an application actually executes. - Agent components and tool setup: Describe the components and configuration used to create the section’s CodeAgent, including its model, toolbox, custom tools, and run method.
Why it matters: Clear tool definitions and an explicit toolbox determine which actions the agent can select and perform. - Flight booking tool workflow: Trace the local flight-booking example from a user task through flight search, seat selection, and mock booking. Explain what the unrelated Fibonacci tool demonstrates.
Why it matters: The example shows agentic planning as a sequence of selected, executed tools rather than a list of suggestions. - Code versus tool-call actions: Distinguish CodeAgent’s Python actions from ToolCallingAgent’s JSON-like actions and explain the reason the section gives for favoring CodeAgent.
Why it matters: The action format affects how flexibly an agent can carry out its plan. - Local agent design tradeoffs: Explain the section’s guidance for choosing and serving a local agent model and for keeping an agent workflow reliable and economical.
Why it matters: A working agent demo still needs appropriate model evaluation and careful division of work to be dependable.
Key terms: AI agent, reason, act, interact, RPA, smolagents, CodeAgent, ReAct, Python snippets, configured tools
13.5 Summary
Where this is used later
- GraphRAG (chapter 14) · Assumes the reader has encountered RAG as a way to supply external context to a language model.
- RAG + Agentic AI (chapter 14) · combining a domain-specific SLM with RAG
- Long- and short-term memory management (chapter 14) · The use of retrieved context to support an LLM response
- Microsoft’s open source GraphRAG capabilities (chapter 14) · The role of a vector database in an LLM application.
Concepts explained in this chapter
- AI agentAn autonomous software entity that can invoke models or tools as part of a larger system.
- CodeAgentA smolagents agent pattern that generates Python code to invoke tools and solve a task step by step.
- ReActAn agent framework that interleaves reasoning and tool use to solve tasks by iterating toward a solution.
- smolagentsA Hugging Face library for building AI agents with code-based or tool-calling workflows.
- vector databaseA database designed to store vectors and support similarity search, typically using approximate nearest neighbor methods for retrieval.
- retrieval-augmented generationAn application system that combines retrieval from external sources with text generation to ground outputs in outside knowledge.
- sentence-transformersA library for producing dense text vectors with transformer encoders for embedding and similarity search.
- knowledge baseRetrieved outside information supplied to a model at inference time to ground its output.
- similarity searchThe process of searching external sources to find information relevant to a query or prompt.
- vector indexA data structure that accelerates retrieval by organizing vectors for efficient similarity search.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.