liveBook · Manning

Agentic RAG

As explained in Retrieval Augmented Generation, The Foundational Ideas

Agentic RAG is a retrieval-augmented generation architecture in which runtime decisions control retrieval, reasoning, or tool actions based on system state and signals.

A single-pass RAG pipeline retrieves once and generates an answer. If it finds a generic policy instead of the specific rule needed, the answer may sound confident but be wrong. Agentic RAG gives a model control over the next step: it can assess the evidence, search again, change its query, or stop. section 11.0 section 11.1

The architectural boundary is who chooses the route. A modular pipeline may have routers, retries, and critique, but its possible paths are specified in advance. In agentic RAG, the model chooses its next action during execution; the number and sequence of steps can change as evidence arrives. This is a stricter distinction than calling any pipeline with a fallback “agentic.” section 1.5 section 11.5.1

The amount of runtime control can vary:

  • A single evaluation and fallback can correct one poor retrieval without an open-ended loop, as in CRAG. section 11.5.1
  • A plan–evaluate–replan loop decomposes a question, retrieves evidence, checks what is missing, and searches again; LevelRAG illustrates this fuller pattern. section 11.5.1
  • A tool-using research agent can choose actions beyond retrieval, including calls to structured-data APIs or code execution. section 11.5.1

A compliance question with moving information needs

“Can we onboard a politically exposed person from the EU using expedited KYC, and what approvals are required?” Answering requires internal policy, external regulation, and an approval matrix. Finding only a general policy is not enough: the system must recognize the missing checks and seek the specific evidence before answering. section 11.1.1 section 11.0

Listing 15.2 Agentic retrieval loop with reflection and replanning.
from openai import OpenAI
 
client = OpenAI()
MODEL = "gpt-4o-mini"
MAX_ITERATIONS = 3  #A
CONFIDENCE_THRESHOLD = 0.7  #B
 
def agentic_rag_loop(
    query: str,
    retrieve_fn: callable,
    k: int = 5
) -> dict:
    """Run an agentic RAG loop with reflection."""
    context = retrieve_fn(query, k=k)  #C
    for iteration in range(MAX_ITERATIONS):
        response = client.chat.completions.create(
            model=MODEL,
            messages=[
                {"role": "system", "content": (
                    "Answer based on the context. After your "
                    "answer, rate your confidence 0-1 and list "
                    "any missing information."
                )},
                {"role": "user", "content": (
                    f"Query: {query}\nContext: {context}"
                )},
            ],
        )
        answer = response.choices[0].message.content
        evaluation = _evaluate_response(query, answer)  #D
        if evaluation["confidence"] >= CONFIDENCE_THRESHOLD:
            return {"answer": answer, "iterations": iteration + 1}
        query = _reformulate_query(query, evaluation["gaps"])
        new_context = retrieve_fn(query, k=k)
        context = context + "\n" + new_context  #E
    return {"answer": answer, "iterations": MAX_ITERATIONS}
 

The small Python loop retrieves context, generates an answer, evaluates confidence and missing information, then either returns or reformulates the query. Its three-iteration cap prevents an unanswerable question from causing endless searches; production evaluation should use quality measures rather than rely only on the model’s self-reported confidence. section 15.2.3

Buy autonomy for observed failures, not for complexity alone. Each evaluation adds a model call, and each reformulation adds retrieval latency. In an offline fintech comparison, retrieval accuracy rose from 54% to 62%, while average latency rose from 0.79 to 5.02 seconds. A well-tuned fixed pipeline can also outperform an agent at some retrieval decisions. Measure a single-pass baseline first, then add only the control needed to fix its failures. section 15.2.4 section 11.5.1

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Defines runtime control, distinguishes it from fixed routing, and implements a capped retrieval–evaluation loop; leaves evaluation helpers outside the shown listing.
Examples
Uses a multi-source compliance question, a Python retrieval loop, and an offline fintech comparison to expose benefits and costs.
What it assumes
Assumes familiarity with basic retrieve-then-generate RAG and Python functions; introduces the feedback-loop architecture without requiring a multi-agent framework.
What you can do after
Build it from scratch as a capped retrieval loop, then evaluate whether its accuracy gain justifies extra latency against a fixed-pipeline baseline.
How it connects
Connects retrieval-augmented generation to query reformulation, retrieval evaluation, planning, self-correction, and tool use.

Also explained in

Compare how 9 books explain agentic RAG →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.