Inference-time compute scaling

As explained in Build a Reasoning Model (From Scratch)

Improving a trained model's output quality by spending more computation at inference time without changing its weights.

Inference-time compute scaling improves a trained model’s answer quality by spending more computation per question without changing its weights. It trades higher accuracy for longer runtime, greater compute use, and often higher API cost. section 4.1 section 4.3

Use three complementary inference-time strategies:

• Extend the response with chain-of-thought prompting, such as “Explain step by step,” so the model writes intermediate reasoning before its final answer. section 4.1 section 4.3
• Generate multiple candidate answers with temperature and top-p sampling, then use self-consistency to select the most frequent final result. section 4.7 section 4.4
• Iteratively critique and improve one answer through self-refinement, using scoring signals such as token log probabilities to track progress. section 5.0 section 5.1

Solving a MATH-500 equation

A base model initially produces the wrong result for “Half the value of 3x−9 is x+37.” Adding “Explain step by step” elicits intermediate algebra and changes the final answer to 83. The longer response improves reasoning but requires more generated tokens. section 4.2 section 4.3

Figure 4.1 A model of the topics covered in this book. This chapter focuses on inference-time-scaling techniques, which improve reasoning without additional training (stage 3). In particular, it extends the text generation function and implements a voting-based method to improve answer accuracy. The next chapter will introduce another inference-time-scaling approach where the model iteratively refines its own answers.

Use the workflow to distinguish prompt extension, voting over sampled answers, and iterative self-refinement within the inference stage. section 4.0

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Explains prompting, sampling, voting, and refinement workflows, while stopping short of weight updates or rule-based internal logic.

Examples

Demonstrates MATH-500 algebra, step-by-step prompting, multiple sampled answers, majority voting, and iterative answer refinement.

What it assumes

Assumes a pretrained language model, tokenizer, prompt formatting, and basic token-generation mechanics; introduces sampling controls and scoring progressively.

What you can do after

Build it from scratch with a flexible generation wrapper, sampling controls, self-consistency voting, and self-refinement scoring.

How it connects

Connects chain-of-thought prompting, self-consistency, self-refinement, temperature scaling, top-p sampling, reinforcement learning, and distillation.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackInference-time scaling improves model performance by allocating additional computation during inference, such as generating and evaluating more candidate reasoning paths.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentInference-time scaling is the allocation of additional computation or generated reasoning during inference to improve a model's output.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionInference-time scaling is increasing computation or deliberation during inference to improve a model's response quality.

Compare how 4 books explain inference-time compute scaling →

Related concepts

• language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
• multi-step reasoninguses · Multi-step reasoning derives a conclusion, solution, or decision through multiple connected intermediate inferential or problem-solving steps.
• tokenuses · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
• trajectoryuses · A trajectory is an ordered sequence of states, actions, events, or operations that records or describes progression through successive steps or over time.
• chain-of-thought promptinguses · Chain-of-thought prompting is a technique that elicits or guides a model to produce or follow intermediate reasoning steps before a final answer.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.