A reflection pattern makes an agent evaluate its own response or intermediate result, identify gaps, and revise through additional retrieval or generation. It targets incomplete answers, unsupported claims, missing details, and inconsistencies with retrieved context. Self-RAG reflection tokens provide a lightweight implementation, while a full agent can use heavier evaluation or an external critic model. section 15.2.1
A practical reflection loop separates response production from response review:
- Retrieve context and generate an initial response. section 15.2.3
- Evaluate completeness, confidence, relevance, and missing information. section 15.2.1 section 15.2.3
- Reformulate the query toward identified gaps, retrieve again, and regenerate when the response is insufficient. section 15.2.3
- Stop when the response meets the threshold, or enforce a maximum iteration count to prevent runaway loops on unanswerable queries. section 15.2.3
Listing 15.2 Agentic retrieval loop with reflection and replanning.
from openai import OpenAI
client = OpenAI()
MODEL = "gpt-4o-mini"
MAX_ITERATIONS = 3 #A
CONFIDENCE_THRESHOLD = 0.7 #B
def agentic_rag_loop(
query: str,
retrieve_fn: callable,
k: int = 5
) -> dict:
"""Run an agentic RAG loop with reflection."""
context = retrieve_fn(query, k=k) #C
for iteration in range(MAX_ITERATIONS):
response = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": (
"Answer based on the context. After your "
"answer, rate your confidence 0-1 and list "
"any missing information."
)},
{"role": "user", "content": (
f"Query: {query}\nContext: {context}"
)},
],
)
answer = response.choices[0].message.content
evaluation = _evaluate_response(query, answer) #D
if evaluation["confidence"] >= CONFIDENCE_THRESHOLD:
return {"answer": answer, "iterations": iteration + 1}
query = _reformulate_query(query, evaluation["gaps"])
new_context = retrieve_fn(query, k=k)
context = context + "\n" + new_context #E
return {"answer": answer, "iterations": MAX_ITERATIONS}
A minimal Python orchestrator demonstrates reflection with an OpenAI model, confidence threshold, retrieval callback, query reformulation, and bounded retries. Replace self-reported confidence with RAGAS faithfulness and relevance scores when evaluating production quality. section 15.2.3
