Iterative self-refinement improves one model answer through repeated inference rather than additional training. The model generates a draft, critiques it, proposes fixes, and generates a revision; an optional scorer keeps the revision only when its score does not decrease. section 5.0 section 5.9
Implement the procedure as a repeatable inference workflow:
- Generate an initial response from the original prompt. section 5.7
- Ask the model to identify logical errors, missing steps, or arithmetic mistakes and provide a concise fix plan. section 5.7
- Give the original question, draft, and critique to a refinement prompt that requests a revised answer. section 5.7
- Repeat for a fixed number of iterations and optionally compare heuristic or average-logprob scores before accepting each revision. section 5.8
Correcting an algebra answer
A model initially returns x = 18 for “Half the value of 3x − 9 is x + 37.” Critique and a fix plan derive ½(3x − 9) = x + 37, yielding x = 83; the refinement produces the boxed result 83.
Figure 5.20 The self-refinement process. The LLM first produces an initial answer to the prompt, and then receives a critique prompt that asks it to analyze its own response and produce a short critique with a plan to refine the answer. In the final step, the model is given a refine prompt that contains the original question, its draft answer, and the critique, and it generates a revised answer that incorporates the suggested improvements.
Follow the three-stage workflow from draft generation through self-critique and revision, including the prompts and information passed between stages. section 5.7
Scoring is useful but not universally beneficial. On MATH-500, one refinement improved the base model from 15.2% to 25.0% without scoring, while two iterations fell to 22.0%; results varied across heuristic and average-logprob scoring and across model types. section 5.8
