Chapter 5: Inference-time scaling via self-refinement
Introduces self-refinement as an inference-time scaling technique and contrasts it with self-consistency.
After this chapter you will be able to
- Self-refinement versus self-consistency. Distinguish self-refinement from self-consistency: self-consistency generates multiple answers and selects the most frequent, while self-refinement iteratively improves a single answer to correct potential mistakes. Explain that both aim to improve response accuracy without further training.
- Self-refinement and scoring roles. Explain the basic idea of self-refinement and what scoring contributes, without describing an implementation of the loop.
- Response quality and cost. Explain why the shorter of two correct responses is not automatically better, and when its lower generation cost makes it preferable.
- Scoring Versus Correctness Checking. Distinguish the heuristic scorer’s ranking of responses by format and brevity from verifying whether an answer is correct. Explain why this section uses a scorer rather than the chapter 3 verifier.
- Extractability Bonus Priorities. Describe how the heuristic rewards a boxed final answer, a merely extractable number, and brevity, and why the relative sizes of those rewards matter.
- Next-token probability scores. Explain how the model’s logits become probabilities for candidate next tokens and how those probabilities are used to compare proposed answers.
- Shifted next-token targets. Trace how the function retrieves the probability of each actual next token in a completed text.
- Joint sequence probability. Explain how individual next-token probabilities form a score for a completed sequence and why the raw result can be tiny.
- Retrospective scoring versus generation. Distinguish scoring an existing answer from greedy or temperature-plus-top-p generation, including why greedy choices need not maximize whole-sequence probability.
- Confidence and correctness distinction. Explain what a higher model-assigned answer score does—and does not—establish.
- Log Probability Interpretation. Explain how token probabilities become log probabilities and how to compare the resulting scores.
- Stable Sequence Log Likelihood. Explain why sequence scoring sums token log probabilities instead of multiplying token probabilities.
- Answer-only average logprob score. Explain how avg_logprob_answer scores a response by excluding prompt tokens and averaging the answer tokens’ log probabilities rather than summing them. Describe why averaging makes comparisons between answers of different lengths fairer.
- Limits of confidence scoring. Explain what a higher answer logprob score indicates in the section’s example and why it cannot be treated as proof that an answer is correct.
- Self-refinement prompt sequence. Trace how the model produces an initial answer, critiques it using a prompt containing the original question and draft, and then revises the answer. Explain that these are sequential uses of text generation and that the loop can be repeated.
- Iterative answer acceptance. Trace how the self-refinement loop handles repeated revisions, including its acceptance rule with and without a scoring function.
- Benchmark refinement outcomes. Compare the reported MATH-500 outcomes for the base and reasoning models, and relate self-refinement to self-consistency on this task.
- Logprob scoring limitations. Explain why average logprob scoring can select a wrong revision despite assigning it a higher score, and distinguish its focus from heuristic scoring.
What it builds on
- Improving response accuracy with self-consistency (chapter 4) · Assumes the earlier introduction to self-consistency as a way to select among multiple answers.
- Generating better responses with chain-of-thought prompting (chapter 4) · The earlier introduction to chain-of-thought prompting.
- Adding temperature scaling to the text generation function (chapter 4) · The temperature-scaled text generation function reused in this check.
- Adding a top-p filter to the text generation function (chapter 4) · The top-p sampling filter reused in this check.
- Grading answers (chapter 3) · The distinction assumes the earlier use of a verifier to grade answers.
- Sampling the next token from a probability distribution (chapter 4) · Softmax probabilities used when sampling a next token.
- Coding a minimal text generation function (chapter 2) · Running token IDs through a model to obtain logits.
- Understanding the sequential LLM text generation process (chapter 2) · Sequential next-token prediction conditioned on preceding text.
Section by section
Introduction
Introduces self-refinement as an inference-time scaling technique and contrasts it with self-consistency.
- Self-refinement versus self-consistency: Distinguish self-refinement from self-consistency: self-consistency generates multiple answers and selects the most frequent, while self-refinement iteratively improves a single answer to correct potential mistakes. Explain that both aim to improve response accuracy without further training.
Why it matters: This distinction explains why the chapter turns from selecting an answer to improving one.
Key terms: inference-time scaling, self-consistency, self-refinement
5.1 Scoring and iteratively improving model responses
The section contrasts earlier inference-time methods with self-refinement and introduces scoring as a way to compare answers and track improvement.
- Chain-of-thought prompting fit: Explain how chain-of-thought prompting can improve a base model’s answers and why it is less useful for models already trained to explain their answers.
Why it matters: It clarifies when asking for step-by-step reasoning adds value. - Self-consistency voting limitation: Describe how self-consistency selects an answer and why its voting approach does not suit every response.
Why it matters: The limitation of majority voting motivates considering other ways to compare responses. - Self-refinement and scoring roles: Explain the basic idea of self-refinement and what scoring contributes, without describing an implementation of the loop.
Why it matters: It connects iterative answer improvement with a way to compare responses and monitor progress.
Key terms: chain-of-thought prompting, base model, reasoning models, self-consistency, most frequent, latency, self-refinement, scoring functions, logprob scoring
5.2 Loading a pretrained model
The section checks a loaded model with sampled math responses, explains why the examples start on CPU, and weighs response quality against generation cost.
- Response quality and cost: Explain why the shorter of two correct responses is not automatically better, and when its lower generation cost makes it preferable.
Why it matters: It separates judging an answer’s value from judging its generation cost. - CPU choice for reproducibility: Justify starting the examples on CPU rather than MPS or CUDA.
Why it matters: Starting on CPU makes small numerical mismatches with the book’s examples less distracting. - Seed-dependent sampled responses: Describe what changing the random seed demonstrates in the model check.
Why it matters: The example shows why two outputs from the same model and prompt can differ even when their final answers agree.
Key terms: qualitative value, tokens, intermediate steps, CPU, MPS, CUDA, temperature scaling, top-p sampling, random seed, MATH-500
5.3 Scoring LLM responses with a rule-based score
The section builds a rule-based scorer that ranks LLM responses by extractable answer format and brevity, then shows how to use and adjust it.
- Scoring Versus Correctness Checking: Distinguish the heuristic scorer’s ranking of responses by format and brevity from verifying whether an answer is correct. Explain why this section uses a scorer rather than the chapter 3 verifier.
Why it matters: A higher heuristic score should not be mistaken for proof that an answer is right. - Extractability Bonus Priorities: Describe how the heuristic rewards a boxed final answer, a merely extractable number, and brevity, and why the relative sizes of those rewards matter.
Why it matters: The relative rewards determine whether the scorer favors a clear final answer over a superficially shorter response. - Character-Based Brevity Reward: Explain how answer length affects the brevity contribution and why the example scorer measures length in characters.
Why it matters: The length term explains why two similarly formatted responses can receive different scores. - Candidate Selection And Tuning: Apply the heuristic as a tiebreaker for equal self-consistency votes or as the selector among N generated candidates, and describe how observed selection problems guide bonus adjustments.
Why it matters: These uses show how a response score becomes a selection rule, and why its settings need to match the failures observed.
Key terms: rule-based scoring function, scorer, verifier, heuristic_score, boxed_bonus, extract_bonus, brevity_bonus, fulltext_bonus, exponentially decaying term, characters
5.4 Understanding token probability scores
This section shows how to score a completed answer using the model’s next-token probabilities, combine them into a sequence score, and interpret that score without mistaking it for correctness.
- Next-token probability scores: Explain how the model’s logits become probabilities for candidate next tokens and how those probabilities are used to compare proposed answers.
Why it matters: These per-token values are the starting point for scoring an entire answer. - Shifted next-token targets: Trace how the function retrieves the probability of each actual next token in a completed text.
Why it matters: The position-to-next-token alignment determines which probabilities the function actually scores. - Joint sequence probability: Explain how individual next-token probabilities form a score for a completed sequence and why the raw result can be tiny.
Why it matters: This connects token scores to whole-answer comparisons and explains a limitation of raw likelihoods. - Retrospective scoring versus generation: Distinguish scoring an existing answer from greedy or temperature-plus-top-p generation, including why greedy choices need not maximize whole-sequence probability.
Why it matters: It prevents confusing an answer’s measured model probability with the process that generated it. - Confidence and correctness distinction: Explain what a higher model-assigned answer score does—and does not—establish.
Why it matters: Model preference can help compare candidates, but it cannot serve as a correctness check by itself. - Probability and likelihood terminology: Distinguish the normalized next-token probabilities from the quantity often called sequence likelihood.
Why it matters: The distinction helps interpret what the function returns despite the book’s use of both terms.
Key terms: logits, torch.softmax, next-token probabilities, calc_next_token_probas, t_idx, next_ids, all_probas[t_idx, next_ids], conditional probabilities, joint probability, sequence likelihood
5.5 From token probability scores to log probabilities
This section turns token probabilities into log probabilities so responses can be scored and compared without multiplying many tiny numbers.
- Log Probability Interpretation: Explain how token probabilities become log probabilities and how to compare the resulting scores.
Why it matters: It lets you read a negative log-probability score without reversing which token or response is more likely. - Stable Sequence Log Likelihood: Explain why sequence scoring sums token log probabilities instead of multiplying token probabilities.
Why it matters: Summing in log space keeps sequence scores useful when ordinary probability products become extremely small. - Matching Log Scoring Operations: Describe the two operation changes used to implement sequence log-probability scoring.
Why it matters: The code must combine scores in the way that matches their mathematical meaning.
Key terms: log probabilities, natural logarithm, logits, joint probability, sequence log likelihood, log space, underflow, torch.log_softmax, torch.softmax, torch.sum
5.6 Scoring model confidence with log probabilities
This section turns token log probabilities into an answer-only, length-normalized score for comparing responses, while explaining why that score is not a guarantee of correctness.
- Answer-only average logprob score: Explain how avg_logprob_answer scores a response by excluding prompt tokens and averaging the answer tokens’ log probabilities rather than summing them. Describe why averaging makes comparisons between answers of different lengths fairer.
Why it matters: This gives you a way to compare candidate answers without including the shared prompt or simply penalizing longer answers. - Limits of confidence scoring: Explain what a higher answer logprob score indicates in the section’s example and why it cannot be treated as proof that an answer is correct.
Why it matters: A model’s confidence can help rank responses, but it should not be mistaken for a correctness check.
Key terms: avg_logprob_answer, answer tokens, token log probabilities, logprob scoring, confident mistakes, scoring signal
5.7 Self-refinement through iterative feedback
The section walks through self-refinement: an LLM generates an answer, critiques it, and uses a refinement prompt to revise it.
- Self-refinement prompt sequence: Trace how the model produces an initial answer, critiques it using a prompt containing the original question and draft, and then revises the answer. Explain that these are sequential uses of text generation and that the loop can be repeated.
Why it matters: The prompt sequence is the basis for understanding the automated self-refinement loop. - Value of imperfect critiques: Explain why a critique can still help self-refinement even when part of it is factually wrong, using the section’s example.
Why it matters: A useful critique should be judged by whether it helps the revision, not by whether every statement in it is right.
Key terms: self-refinement, critique prompt, refinement prompt, draft, refinement loop, critique
5.8 Coding the self-refinement loop
This section turns self-refinement into an iterative function with optional scoring, then uses benchmark results to show when refinement and its scorers help or hurt.
- Iterative answer acceptance: Trace how the self-refinement loop handles repeated revisions, including its acceptance rule with and without a scoring function.
Why it matters: The acceptance rule determines whether the loop keeps an earlier answer or replaces it with a revision. - Logprob scorer integration: Describe how the average logprob scorer is made compatible with the loop and what the returned results let a caller inspect.
Why it matters: Adapting the scorer makes an existing scoring function usable in the loop without changing its interface. - Benchmark refinement outcomes: Compare the reported MATH-500 outcomes for the base and reasoning models, and relate self-refinement to self-consistency on this task.
Why it matters: The benchmark cautions against assuming that more refinement or added scoring will reliably produce better answers. - Logprob scoring limitations: Explain why average logprob scoring can select a wrong revision despite assigning it a higher score, and distinguish its focus from heuristic scoring.
Why it matters: A higher model-confidence score is not proof that a revised mathematical answer is correct.
Key terms: self_refinement_loop, iterations, score_fn, revised_score, current_score, avg_logprob_answer, functools.partial, final_extracted, final_full, MATH-500
5.9 Summary
The section recaps how self-refinement revises an answer and how scoring can evaluate responses and guard against worse revisions.
Where this is used later
- Summary (chapter 6) · Log probabilities used to score generated tokens
- Entropy tracking (chapter 7) · Log probabilities of model tokens.
- Scoring rollouts with sequence log probabilities (chapter 6) · Scoring rollouts with sequence log probabilities.
Concepts explained in this chapter
- log probabilityThe logarithm of a model-assigned probability, used here to score how likely a generated token or whole sequence is under the current parameters.
- token probabilityThe probability assigned by the model to a candidate token at a particular position in the output sequence.
- continuous improvementAn inference-time procedure that iteratively revises a model answer and keeps a revision only if it scores better than the current answer.
- joint probability distributionThe probability of an entire token sequence under a model, computed by multiplying conditional token probabilities across positions.
- softmaxA normalization function that converts logits into a probability distribution over possible next tokens.
- confidenceThe model-assigned probability used here as a scorer for how strongly an LLM prefers a candidate answer.
- torch.prodPyTorch's product operation used here to multiply per-token probabilities into a sequence score.
- logprob scorerA scoring method that ranks answers by their average log probability under the model, favoring fluent and likely text.
- numerical stabilityThe property of a computation remaining accurate and avoiding underflow or rounding errors when values are very small or very large.
- torch.log_softmaxA PyTorch function that computes log-softmax over logits to produce log probabilities in a numerically stable way.
- rule-based scoring functionA manually specified scoring method that assigns numeric values to model outputs using explicit rules rather than learned parameters.
- inference-time compute scalingImproving a trained model's output quality by spending more computation at inference time without changing its weights.
- self-consistencyAn inference technique that generates multiple candidate responses and selects or combines them to improve answer quality.
- reinforcement learning with verifiable rewardsA reinforcement learning approach that uses automatically checkable correctness rewards instead of human preference signals.
- generationThe process of producing new textual output from a language model given prior context or instructions.
← Chapter 4: Improving reasoning with inference-time scaling
Chapter 6: Training reasoning models with reinforcement learning →
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.