liveBook · Manning

Reflection pattern

As explained in Retrieval Augmented Generation, The Foundational Ideas

A reflection pattern is an agentic control pattern in which a system evaluates its own output or intermediate results and revises them when needed.

A reflection pattern makes an agent evaluate its own response or intermediate result, identify gaps, and revise through additional retrieval or generation. It targets incomplete answers, unsupported claims, missing details, and inconsistencies with retrieved context. Self-RAG reflection tokens provide a lightweight implementation, while a full agent can use heavier evaluation or an external critic model. section 15.2.1

A practical reflection loop separates response production from response review:

  • Retrieve context and generate an initial response. section 15.2.3
  • Evaluate completeness, confidence, relevance, and missing information. section 15.2.1 section 15.2.3
  • Reformulate the query toward identified gaps, retrieve again, and regenerate when the response is insufficient. section 15.2.3
  • Stop when the response meets the threshold, or enforce a maximum iteration count to prevent runaway loops on unanswerable queries. section 15.2.3
Listing 15.2 Agentic retrieval loop with reflection and replanning.
from openai import OpenAI
 
client = OpenAI()
MODEL = "gpt-4o-mini"
MAX_ITERATIONS = 3  #A
CONFIDENCE_THRESHOLD = 0.7  #B
 
def agentic_rag_loop(
    query: str,
    retrieve_fn: callable,
    k: int = 5
) -> dict:
    """Run an agentic RAG loop with reflection."""
    context = retrieve_fn(query, k=k)  #C
    for iteration in range(MAX_ITERATIONS):
        response = client.chat.completions.create(
            model=MODEL,
            messages=[
                {"role": "system", "content": (
                    "Answer based on the context. After your "
                    "answer, rate your confidence 0-1 and list "
                    "any missing information."
                )},
                {"role": "user", "content": (
                    f"Query: {query}\nContext: {context}"
                )},
            ],
        )
        answer = response.choices[0].message.content
        evaluation = _evaluate_response(query, answer)  #D
        if evaluation["confidence"] >= CONFIDENCE_THRESHOLD:
            return {"answer": answer, "iterations": iteration + 1}
        query = _reformulate_query(query, evaluation["gaps"])
        new_context = retrieve_fn(query, k=k)
        context = context + "\n" + new_context  #E
    return {"answer": answer, "iterations": MAX_ITERATIONS}
 

A minimal Python orchestrator demonstrates reflection with an OpenAI model, confidence threshold, retrieval callback, query reformulation, and bounded retries. Replace self-reported confidence with RAGAS faithfulness and relevance scores when evaluating production quality. section 15.2.3

How Retrieval Augmented Generation, The Foundational Ideas covers it

How deep it goes
Explains the retrieve-generate-evaluate loop and bounded refinement, but defers detailed evaluator implementation and uses self-reported confidence in the example.
Examples
Uses Self-RAG reflection tokens and a Python agentic loop that revises queries when confidence or completeness is inadequate.
What it assumes
Assumes retrieval, generation, and basic RAG evaluation; teaches the control loop without requiring a multi-agent framework.
What you can do after
Implement a bounded reflection loop that evaluates responses, targets gaps with reformulated queries, and measures quality with RAGAS.
How it connects
Connects reflection with Self-RAG, retrieval, generation, planning, query reformulation, and RAGAS evaluation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Retrieval Augmented Generation, The Foundational Ideas. The section links point to the parts of the book it draws on.