Inference-time compute scaling

As explained in Reinforcement Learning from Human Feedback

Inference-time scaling improves model performance by allocating additional computation during inference, such as generating and evaluating more candidate reasoning paths.

Inference-time compute scaling, or test-time scaling, improves downstream performance by spending more computation while generating an answer. A model may generate a longer reasoning chain, sample several candidate responses, or make additional inference calls rather than committing to its first output. The technique matters when extra tokens or rollouts produce better task performance rather than merely consuming energy. section 7.0 section 7.2.2

Reasoning-trained models are especially associated with this behavior. Reinforcement learning with verifiable rewards (RLVR) can elicit reasoning that uses more generated tokens, and longer reasoning chains are strongly correlated with improved results on verifiable downstream tasks. This differs from early RLHF length bias: increasing response length for small preference-ranking gains is not itself useful scaling. The operational test is whether added inference compute improves the final answer. section 7.2.2 section 7.5

Common ways to allocate extra inference compute include:

• Generate a longer reasoning process before producing the answer. section 7.0
• Sample repeated responses and extract an answer from the resulting candidates. section 7.2.2
• Run parallel, independently sampled rollouts and return the majority answer. section 7.3.3
• Score parallel candidates with a selection model and choose the best answer rather than relying on a vote. section 7.3.3
• Use value-guided sampling or compose additional inference calls to search more deliberately. section 7.2.2

Parallel rollout selection

For a question with a verifiable answer, sample several independent reasoning rollouts. A basic pipeline extracts each final answer and returns the majority result. A more advanced pipeline sends the candidate answers to a scoring model, which selects the strongest one. Both pipelines spend more inference compute than a single rollout; the latter replaces agreement with learned selection. section 7.3.3

Compute budgets can also be controlled. Progressive length scheduling trains effective reasoning under a limited thinking budget before extending it to harder problems, helping limit overthinking. Some models expose reasoning-length controls through system prompts or low–medium–high settings. Distillation from a larger RL-trained model to a smaller one is another explored route, but the broader method landscape is evolving quickly. section 7.3.3 section 7.2.2

Evaluation must account for the inference budget. Since models can gain performance by using more tokens, comparing scores without controlling total inference tokens can confound model capability with compute allocation. Reproducible comparisons are further complicated by hidden prompts, tokens, evaluation configurations, and output-format sensitivity. section 16.2

Figure 7.1 RLVR in the form of an RL feedback loop. Instead of a reward model, a verification function is used.

The RLVR feedback-loop diagram situates inference-time scaling beside the training process that commonly elicits reasoning behavior: verification functions replace a learned reward model. section 7.0

How Reinforcement Learning from Human Feedback covers it

How deep it goes

Defines compute-allocation methods and rollout selection, while deferring most rapidly evolving techniques.

Examples

Uses longer reasoning chains, repeated sampling, majority-voted parallel rollouts, and scoring-model selection.

What it assumes

Assumes familiarity with language-model generation and introduces scaling through concrete sampling and reasoning behaviors.

What you can do after

Evaluate/understand when extra tokens or rollouts improve answers and when they are wasted energy.

How it connects

Connects inference budgets to RLVR, RLHF, chain-of-thought reasoning, value-guided sampling, and evaluation.

liveBookmanning.com

Read Reinforcement Learning from Human Feedback on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Build a Reasoning Model (From Scratch)Improving a trained model's output quality by spending more computation at inference time without changing its weights.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentInference-time scaling is the allocation of additional computation or generated reasoning during inference to improve a model's output.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionInference-time scaling is increasing computation or deliberation during inference to improve a model's response quality.

Compare how 4 books explain inference-time compute scaling →

Related concepts

• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• multi-step reasoninguses · Multi-step reasoning derives a conclusion, solution, or decision through multiple connected intermediate inferential or problem-solving steps.
• trajectoryuses · A trajectory is an ordered sequence of states, actions, events, or operations that records or describes progression through successive steps or over time.
• tokenuses · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
• chain-of-thought promptinguses · Chain-of-thought prompting is a technique that elicits or guides a model to produce or follow intermediate reasoning steps before a final answer.

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.