A single-pass RAG pipeline retrieves once and generates an answer. If it finds a generic policy instead of the specific rule needed, the answer may sound confident but be wrong. Agentic RAG gives a model control over the next step: it can assess the evidence, search again, change its query, or stop. section 11.0 section 11.1
The architectural boundary is who chooses the route. A modular pipeline may have routers, retries, and critique, but its possible paths are specified in advance. In agentic RAG, the model chooses its next action during execution; the number and sequence of steps can change as evidence arrives. This is a stricter distinction than calling any pipeline with a fallback “agentic.” section 1.5 section 11.5.1
The amount of runtime control can vary:
- A single evaluation and fallback can correct one poor retrieval without an open-ended loop, as in CRAG. section 11.5.1
- A plan–evaluate–replan loop decomposes a question, retrieves evidence, checks what is missing, and searches again; LevelRAG illustrates this fuller pattern. section 11.5.1
- A tool-using research agent can choose actions beyond retrieval, including calls to structured-data APIs or code execution. section 11.5.1
A compliance question with moving information needs
“Can we onboard a politically exposed person from the EU using expedited KYC, and what approvals are required?” Answering requires internal policy, external regulation, and an approval matrix. Finding only a general policy is not enough: the system must recognize the missing checks and seek the specific evidence before answering. section 11.1.1 section 11.0
Listing 15.2 Agentic retrieval loop with reflection and replanning.
from openai import OpenAI
client = OpenAI()
MODEL = "gpt-4o-mini"
MAX_ITERATIONS = 3 #A
CONFIDENCE_THRESHOLD = 0.7 #B
def agentic_rag_loop(
query: str,
retrieve_fn: callable,
k: int = 5
) -> dict:
"""Run an agentic RAG loop with reflection."""
context = retrieve_fn(query, k=k) #C
for iteration in range(MAX_ITERATIONS):
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": (
"Answer based on the context. After your "
"answer, rate your confidence 0-1 and list "
"any missing information."
)},
{"role": "user", "content": (
f"Query: {query}\nContext: {context}"
)},
],
)
answer = response.choices[0].message.content
evaluation = _evaluate_response(query, answer) #D
if evaluation["confidence"] >= CONFIDENCE_THRESHOLD:
return {"answer": answer, "iterations": iteration + 1}
query = _reformulate_query(query, evaluation["gaps"])
new_context = retrieve_fn(query, k=k)
context = context + "\n" + new_context #E
return {"answer": answer, "iterations": MAX_ITERATIONS}
The small Python loop retrieves context, generates an answer, evaluates confidence and missing information, then either returns or reformulates the query. Its three-iteration cap prevents an unanswerable question from causing endless searches; production evaluation should use quality measures rather than rely only on the model’s self-reported confidence. section 15.2.3
Buy autonomy for observed failures, not for complexity alone. Each evaluation adds a model call, and each reformulation adds retrieval latency. In an offline fintech comparison, retrieval accuracy rose from 54% to 62%, while average latency rose from 0.79 to 5.02 seconds. A well-tuned fixed pipeline can also outperform an agent at some retrieval decisions. Measure a single-pass baseline first, then add only the control needed to fix its failures. section 15.2.4 section 11.5.1








