An answer critic evaluates whether retriever outputs correctly and completely answer the original question. As a blocking function, it prevents unsupported or incomplete answers from reaching the user. section 5.1 section 5.1.3
Use the critic to control retrieval quality:
- Pass the original question and all retriever answers to an evaluator. section 5.3.3
- Return an empty question list when the supplied information is sufficient. section 5.3.3
- Generate complete, atomic, specific follow-up questions for missing information. section 5.1.3 section 5.3.3
- Stop retrying when the data source cannot provide the answer, and report that the information is unavailable. section 5.1.3
Critique-and-retrieve cycle
A retriever returns an answer missing part of the user’s question. The critic emits targeted follow-up questions, which go back through the retriever router. The system then supplies the expanded answers to the final LLM. A practical integration critiques once; if the result remains incomplete, the final response must disclose the missing information rather than inventing it. section 5.1.3 section 5.3.4
Listing 5.10 Answer critic instructions
answer_critique_prompt = """
You are an expert at identifying if questions have been fully answered or if there is an opportunity to enrich the answer.
The user will provide a question, and you will scan through the provided information to see if the question is answered.
If anything is missing from the answer, you will provide a set of new questions that can be asked to gather the missing information.
All new questions must be complete, atomic, and specific.
However, if the provided information is enough to answer the original question, you will respond with an empty list.
JSON template to use for finding missing information:
{
"questions": ["question1", "question2"]
}
"""
Inspect the JSON prompt pattern that requests an empty list for complete answers and atomic, specific questions for missing information. section 5.3.3
