Chapter 5: Agentic RAG
This introduction frames agentic RAG as a multi-retriever system in which a router selects suitable retrieval agents, typically through LLM tool use.
After this chapter you will be able to
- Agentic RAG architecture. Describe agentic RAG as a system with multiple retrieval agents available to retrieve the data needed for a user question, with a retriever router as its usual entry interface.
- Agentic RAG scope. Explain agentic RAG as a RAG system that can act on a user's behalf by selecting among available retrieval agents to obtain data for answering a question. Distinguish this chapter's basic form, which chooses retrievers and checks whether context answers the question, from more advanced agents that can create task plans.
- Agentic RAG components. Describe the three foundational parts of agentic RAG: a retriever router that selects the best retriever or retrievers for the user's question, retriever agents that retrieve needed data, and an answer critic that checks whether retrieved answers correctly answer the original question.
- Retriever agent role. Describe retriever agents as the components that retrieve the data needed to answer a user's question, and explain that they can range from broad search methods to narrowly defined query templates.
- Specialized retriever rationale. Explain why specialized, narrow retrievers are added over time: generic retrievers may not meet user expectations in production, so recurring question types that they cannot generate effective queries for motivate purpose-built retrievers.
- Retriever router selection. Explain that a retriever router takes a user question and returns the best retriever or retrievers to use, usually using an LLM to make the selection.
- Retriever arguments from questions. Describe how the router can select a retriever and extract the argument required to call it, as when a question about France's capital becomes capital_by_country("France").
- Answer critic validation. Explain that an answer critic receives answers from retrievers and checks whether they correctly and completely answer the original question.
- Blocked-answer recovery loop. Describe how the answer critic blocks an incorrect or incomplete answer, generates a new retrieval question, and initiates another retrieval round; explain that exit criteria are required when the data source does not contain the answer.
- Agentic RAG use cases. Describe the situations in which agentic RAG is useful: when an application has multiple data sources and must choose the best source, or when a broad or complex source needs specialized retrieval.
- Generic and specialized retrieval. Distinguish generic retrievers from specialized retrievers, including vector similarity search for unstructured data and text2cypher for structured graph data, and explain why complex queries can require narrower tools.
- Answer assessment for production. Explain that agentic RAG must both select the best available retriever and assess its answer before returning it, helping production systems maintain high performance and consistent answer quality.
- Retriever tool portfolio. Describe the example set of retriever tools: Cypher-template retrievers for movies by title and actor name, plus a text2cypher retriever for other questions. Explain that retrievers may range from broad searches to narrowly parameterized hardcoded queries and should be added over time as application needs emerge.
- Tool descriptions and parameters. Explain why each retriever tool and its parameters must be described clearly for the LLM. State that these descriptions let the LLM understand a tool, select it appropriately, and supply the right parameter values.
- LLM and system responsibilities. Distinguish the LLM's role from the calling system's role in tool use: the LLM chooses a retriever and parameters, while the external system actually executes the retriever call.
- LLM tool-call handling. Describe how the system executes retriever calls selected by the LLM. Explain that a tool dictionary maps tool names to callable retriever functions, while the LLM returns one or more selected tool calls and their arguments for the system to invoke.
- Sequential query updating. Explain why the router processes questions one at a time and how it rewrites remaining dependent questions using answers from earlier retrievals. State that the query updater receives the original question and retriever answers, returns an updated question in JSON, and must only make the question more complete rather than request new information.
- Retriever routing flow. Trace the retriever router flow from a single question to a retrieval answer. Explain that the router gives the LLM the question, available tools, and prior answers; the LLM selects the best retriever or retrievers through function calling; and the router handles those calls while a surrounding loop updates questions as it proceeds.
- Answer critic responsibility. Explain that the answer critic evaluates all retriever answers against the original user question to determine whether that question has been answered correctly.
- Critic input-output contract. Describe the answer critic function's inputs and outputs: it receives the original question plus retriever answers, calls an LLM using instructions and a JSON format, and can return new questions when the answer is incomplete.
- Grounded answer instruction. Explain why the main LLM is instructed to answer questions using only information supplied in its prompt. State that this constraint supports consistent, trustworthy answers.
- Main agentic RAG flow. Describe the responsibility of the main function: it accepts user input, runs it through the agentic RAG system, and returns an available answer to the user.
- One-time critique policy. Trace the handling of an incomplete or incorrect answer: the critique function produces new questions to collect missing information, but the system critiques only once and then returns any still-incomplete answer with the LLM indicating what is missing.
What it builds on
- The retriever (chapter 2) · The basic role of a retriever in a RAG architecture.
- Performing vector search (chapter 2) · Using vector similarity search to find relevant information in an unstructured text corpus.
- Where query language generation fits in the RAG pipeline (chapter 4) · Using graph-query generation within the RAG pipeline to retrieve structured graph data.
- Components of a RAG architecture (chapter 2) · The distinction between retrieval and answer generation within a RAG architecture.
- The generator (chapter 2) · The generator's role in producing an answer from retrieved information.
- RAG using vector similarity search (chapter 2) · Vector similarity search RAG over a text corpus.
- Implementing a text2cypher generator using a base model (chapter 4) · A text2cypher generator that turns questions into Cypher queries.
- Concluding thoughts (chapter 2) · The limitations and practical considerations of vector and hybrid retrieval.
Section by section
Introduction
This introduction frames agentic RAG as a multi-retriever system in which a router selects suitable retrieval agents, typically through LLM tool use.
- Agentic RAG architecture: Describe agentic RAG as a system with multiple retrieval agents available to retrieve the data needed for a user question, with a retriever router as its usual entry interface.
Why it matters: This architecture lets a RAG application choose among different ways of retrieving information rather than relying on one fixed retriever. - Structured data retrieval value: Explain why structured data can add value beyond unstructured data retrieved with vector similarity search: the structure itself contains information. Describe structure as something that can be added incrementally, from simple to more complex graph data.
Why it matters: The value contained in graph structure motivates having different retrieval approaches available in an agentic RAG system. - LLM tool-use implementation: State that a common implementation of agentic RAG uses an LLM's ability to use tools, also called function calling, and that the chapter uses OpenAI GPT-3.5 and GPT-4 for this capability. Describe ReAct as an approach that can achieve this with most LLMs when native tool use is unavailable.
Why it matters: Tool use is the practical mechanism through which an LLM can invoke the retrievers selected for an agentic RAG task.
Key terms: agentic RAG system, retrieval agents, retriever router, vector similarity search, unstructured data, structured data, graph data, tool use, function calling, GPT-3.5
5.1 What is agentic RAG?
This section defines a basic agentic RAG system and introduces the router, retriever agents, and answer critic that make it work.
- Agentic RAG scope: Explain agentic RAG as a RAG system that can act on a user's behalf by selecting among available retrieval agents to obtain data for answering a question. Distinguish this chapter's basic form, which chooses retrievers and checks whether context answers the question, from more advanced agents that can create task plans.
Why it matters: This scope clarifies what makes the chapter's RAG pipeline agentic without requiring a fully autonomous planning system. - Agentic RAG components: Describe the three foundational parts of agentic RAG: a retriever router that selects the best retriever or retrievers for the user's question, retriever agents that retrieve needed data, and an answer critic that checks whether retrieved answers correctly answer the original question.
Why it matters: These components divide an agentic RAG workflow into choosing a retrieval method, retrieving evidence, and verifying that the question was answered.
Key terms: agentic systems, agentic RAG, retrieval agents, retriever router, retriever agents, answer critic, user question, original question
5.1.1 Retriever agents
This section defines retriever agents, contrasts generic retrieval approaches with specialized ones, and explains why production agentic RAG systems need both.
- Retriever agent role: Describe retriever agents as the components that retrieve the data needed to answer a user's question, and explain that they can range from broad search methods to narrowly defined query templates.
Why it matters: Retriever agents give an agentic RAG system multiple ways to obtain the evidence needed for an answer. - Generic retriever types: Distinguish vector similarity search from text2cypher as generic retriever agents: vector similarity search serves unstructured data sources, whereas text2cypher retrieves structured data from a graph database.
Why it matters: Matching a question's data source to an appropriate generic retriever is necessary before a system can retrieve useful context. - Specialized retriever rationale: Explain why specialized, narrow retrievers are added over time: generic retrievers may not meet user expectations in production, so recurring question types that they cannot generate effective queries for motivate purpose-built retrievers.
Why it matters: Specialized retrievers let a production agentic RAG system improve reliability for question types where general retrieval approaches fall short.
Key terms: retriever agents, vector similarity search, hardcoded database query, query template, text2cypher, unstructured data, structured data, graph database, specialized retrievers, generic retrievers
5.1.2 The retriever router
This section explains how a retriever router uses a user question, often with an LLM, to select the appropriate retriever and supply its needed argument.
- Retriever router selection: Explain that a retriever router takes a user question and returns the best retriever or retrievers to use, usually using an LLM to make the selection.
Why it matters: A router lets an agentic RAG system choose among specialized retrieval capabilities instead of using one retriever for every question. - Retriever arguments from questions: Describe how the router can select a retriever and extract the argument required to call it, as when a question about France's capital becomes capital_by_country("France").
Why it matters: Selecting a tool is not enough: the chosen retriever must also receive the specific value needed to execute its database query.
Key terms: retriever router, retriever agents, LLM, capital_by_country, country_by_capital, extracted argument, hardcoded database queries
5.1.3 Answer critic
This section defines an answer critic that validates retrieved answers, blocks inadequate responses, and drives a bounded retry loop or unavailable-answer response.
- Answer critic validation: Explain that an answer critic receives answers from retrievers and checks whether they correctly and completely answer the original question.
Why it matters: The critic prevents a RAG application from returning an answer that does not actually satisfy the user's question. - Blocked-answer recovery loop: Describe how the answer critic blocks an incorrect or incomplete answer, generates a new retrieval question, and initiates another retrieval round; explain that exit criteria are required when the data source does not contain the answer.
Why it matters: A bounded recovery loop lets the system seek a better answer without endlessly retrying when the information is unavailable.
Key terms: answer critic, retrievers, original question, blocking function, incomplete answer, incorrect answer, new question, exit criteria, data source
5.2 Why do we need agentic RAG?
This section explains why agentic RAG is valuable for selecting among multiple retrievers and checking answers, especially when data sources or queries are complex.
- Agentic RAG use cases: Describe the situations in which agentic RAG is useful: when an application has multiple data sources and must choose the best source, or when a broad or complex source needs specialized retrieval.
Why it matters: Recognizing these situations helps you decide when a single generic RAG retriever is not enough. - Generic and specialized retrieval: Distinguish generic retrievers from specialized retrievers, including vector similarity search for unstructured data and text2cypher for structured graph data, and explain why complex queries can require narrower tools.
Why it matters: This distinction explains why an agent needs a portfolio of retrieval tools rather than relying on one method. - Retriever fallback strategy: Explain how teams can identify recurring questions that text2cypher struggles with, build specialized retrievers for them, and retain text2cypher as a catchall when no specific retriever matches.
Why it matters: This provides a practical way to improve retrieval coverage incrementally rather than attempting to anticipate every difficult query upfront. - Answer assessment for production: Explain that agentic RAG must both select the best available retriever and assess its answer before returning it, helping production systems maintain high performance and consistent answer quality.
Why it matters: Routing alone is insufficient if the application returns answers without checking their quality.
Key terms: agentic RAG, data sources, specialized retrievers, generic retrievers, vector similarity search, unstructured data, graph database, text2cypher, hard-coded database query, catchall retriever
5.3 How to implement agentic RAG
This section introduces the upcoming implementation of an agentic RAG system and points readers to its accompanying notebook and dataset.
- Agentic RAG implementation scope: Describe that the following subsections implement the foundational parts of an agentic RAG system, rather than presenting a complete standalone design in this introductory section.
Why it matters: This frames the code that follows as the foundational implementation of the agentic RAG workflow. - Accompanying Jupyter notebook: State that readers can follow the implementation in the accompanying Chapter 5 Jupyter notebook hosted in the kg-rag GitHub repository.
Why it matters: The notebook provides the executable implementation that accompanies the discussion. - Movies dataset usage: State that this chapter's implementation uses the Movies dataset, and that the appendix provides more information and ways to load that dataset.
Why it matters: Knowing the dataset context helps readers locate the data assumptions behind the implementation.
Key terms: agentic RAG system, foundational parts, Jupyter notebook, kg-rag, ch05.ipynb, Movies dataset, appendix
5.3.1 Implementing retriever tools
This section defines the retriever tools an agentic RAG router can choose from and explains how to describe, invoke, and extend them.
- Retriever tool portfolio: Describe the example set of retriever tools: Cypher-template retrievers for movies by title and actor name, plus a text2cypher retriever for other questions. Explain that retrievers may range from broad searches to narrowly parameterized hardcoded queries and should be added over time as application needs emerge.
Why it matters: A router can select useful retrieval only when the application exposes an appropriate set of retriever options. - Tool descriptions and parameters: Explain why each retriever tool and its parameters must be described clearly for the LLM. State that these descriptions let the LLM understand a tool, select it appropriately, and supply the right parameter values.
Why it matters: The router’s tool choice is only as reliable as its understanding of what each tool does and what inputs it requires. - LLM and system responsibilities: Distinguish the LLM's role from the calling system's role in tool use: the LLM chooses a retriever and parameters, while the external system actually executes the retriever call.
Why it matters: Keeping decision-making separate from execution is essential for implementing a working and controlled agentic RAG pipeline. - Generic context retriever: Describe the generic retriever tool that is used when an answer is already present in the question or surrounding context. Explain that it extracts and returns that existing answer rather than performing external retrieval.
Why it matters: This fallback avoids unnecessary retrieval when the information needed to answer is already available in the prompt or context.
Key terms: retrievers, Cypher templates, text2cypher, vector similarity search, retriever description, parameters, OpenAI's tools format, LLM, retriever, system that calls the LLM
5.3.2 Implementing the retriever router
This section implements a retriever router that uses LLM tool calling to select and invoke retrievers, process dependent questions sequentially, and update later queries with earlier answers.
- LLM tool-call handling: Describe how the system executes retriever calls selected by the LLM. Explain that a tool dictionary maps tool names to callable retriever functions, while the LLM returns one or more selected tool calls and their arguments for the system to invoke.
Why it matters: The LLM chooses retrievers, but application code must perform the actual retriever calls and return their results. - Sequential query updating: Explain why the router processes questions one at a time and how it rewrites remaining dependent questions using answers from earlier retrievals. State that the query updater receives the original question and retriever answers, returns an updated question in JSON, and must only make the question more complete rather than request new information.
Why it matters: Updating follow-up questions with retrieved facts lets later retrieval target the complete question rather than an unresolved reference. - Retriever routing flow: Trace the retriever router flow from a single question to a retrieval answer. Explain that the router gives the LLM the question, available tools, and prior answers; the LLM selects the best retriever or retrievers through function calling; and the router handles those calls while a surrounding loop updates questions as it proceeds.
Why it matters: This flow is the central coordination mechanism that makes the agentic RAG system select retrieval methods dynamically and remain extensible.
Key terms: tool calls, tools, retriever, handle_tool_calls, LLM, continuous query updating, query updater, original question, updated question, retriever answers
5.3.3 Implementing the answer critic
This section implements an LLM-based answer critic that checks retriever results against the original question and requests follow-up retrieval when information is missing.
- Answer critic responsibility: Explain that the answer critic evaluates all retriever answers against the original user question to determine whether that question has been answered correctly.
Why it matters: This check helps ensure the system delivers the information the user asked for rather than trusting intermediate retrieval steps. - Critic input-output contract: Describe the answer critic function's inputs and outputs: it receives the original question plus retriever answers, calls an LLM using instructions and a JSON format, and can return new questions when the answer is incomplete.
Why it matters: A clear input-output contract lets the critic participate reliably in an automated retrieval loop. - Iterative retrieval safeguards: Trace the follow-up loop in which new questions from the critic are sent through the retriever router again, and explain why the loop needs exit criteria.
Why it matters: Without exit criteria, repeated attempts to retrieve unavailable information could leave the application stuck in a loop.
Key terms: answer critic, retrievers, original question, JSON format, answer critic function, new questions, LLM, retriever router, missing information, exit criteria
5.3.4 Tying it all together
This section combines the retriever tools, router, and answer critic into a main agentic RAG function with grounded answering and a one-time critique loop.
- Grounded answer instruction: Explain why the main LLM is instructed to answer questions using only information supplied in its prompt. State that this constraint supports consistent, trustworthy answers.
Why it matters: Grounding the answer in supplied context makes the system's responses more reliable and predictable. - Main agentic RAG flow: Describe the responsibility of the main function: it accepts user input, runs it through the agentic RAG system, and returns an available answer to the user.
Why it matters: The main function is the integration point that turns the separate agents into a usable question-answering application. - One-time critique policy: Trace the handling of an incomplete or incorrect answer: the critique function produces new questions to collect missing information, but the system critiques only once and then returns any still-incomplete answer with the LLM indicating what is missing.
Why it matters: The one-time limit defines how the system attempts to improve an answer without repeatedly looping through critique.
Key terms: instructions, prompt, consistent, trust, main function, user input, agentic RAG system, answer, critique function, new questions
5.4 Summary
This summary recaps agentic RAG as a tool-calling system that routes questions to suitable retriever agents and checks their answers.
Where this is used later
- (chapter intro) (chapter 8) · The answer critic as an agent component concerned with answer quality.
- Next steps (chapter 8) · Implementation of retriever tools and other components of an agentic RAG system
- Coming up with test examples (chapter 8) · Retriever tools as capabilities available to an agent.
Concepts explained in this chapter
- retriever routerA retriever router is a function that selects the most appropriate retriever for a user question and may extract the retriever's input arguments.
- query updaterA query updater is a component that rewrites outstanding questions using the original question and answers obtained from earlier retrievals.
- answer criticAn answer critic is a component that evaluates whether a retrieved answer correctly and completely answers the original question and can prevent unsupported answers from being returned.
- agentic RAGAgentic RAG is a retrieval-augmented generation pattern in which agents and routing mechanisms select and coordinate retrieval actions to answer a question.
- Text2cypherText2cypher is a technique that converts natural-language questions into Cypher queries for graph-database retrieval.
← Chapter 4: Generating Cypher queries from natural language questions
Chapter 6: Constructing knowledge graphs with LLMs →
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.