Inference-time compute scaling
Inference-time compute scaling is allocating additional computation during inference to improve a trained model’s output quality without changing its parameters.
Build a Reasoning Model (From Scratch)
Improving a trained model's output quality by spending more computation at inference time without changing its weights.
- How it is used
- Improves reasoning and answer accuracy during generation when retraining or modifying model weights is undesirable.
- How deep it goes
- Explains prompting, sampling, voting, and refinement workflows, while stopping short of weight updates or rule-based internal logic.
- Examples
- Demonstrates MATH-500 algebra, step-by-step prompting, multiple sampled answers, majority voting, and iterative answer refinement.
- What you can do after
- Build it from scratch with a flexible generation wrapper, sampling controls, self-consistency voting, and self-refinement scoring.
Read how Build a Reasoning Model (From Scratch) explains inference-time compute scaling →
Reinforcement Learning from Human Feedback
Inference-time scaling improves model performance by allocating additional computation during inference, such as generating and evaluating more candidate reasoning paths.
- How it is used
- Improves downstream reasoning performance by allocating additional generation tokens, samples, or inference calls.
- How deep it goes
- Defines compute-allocation methods and rollout selection, while deferring most rapidly evolving techniques.
- Examples
- Uses longer reasoning chains, repeated sampling, majority-voted parallel rollouts, and scoring-model selection.
- What you can do after
- Evaluate/understand when extra tokens or rollouts improve answers and when they are wasted energy.
Read how Reinforcement Learning from Human Feedback explains inference-time compute scaling →
The Art of AI Product Development
Inference-time scaling is the allocation of additional computation or generated reasoning during inference to improve a model's output.
- How it is used
- Improves outputs for complex, multistep language-model tasks by allocating additional generated reasoning during inference.
- How deep it goes
- Explains token-based reasoning, decomposition, verification, refinement, voting, and cost trade-offs without implementation detail.
- Examples
- Contrasts a Vietnam capital question with coding and business-modeling tasks that require extended reasoning.
- What you can do after
- Apply it to coding or business-modeling tasks through decomposition, iterative refinement, or voting when quality justifies added cost.
Read how The Art of AI Product Development explains inference-time compute scaling →
Introduction to Generative AI, Second Edition
Inference-time scaling is increasing computation or deliberation during inference to improve a model's response quality.
- How it is used
- Uses additional deliberation during inference to refine language-model responses after pretraining.
- How deep it goes
- Explains the idea through chain-of-thought prompting but does not detail computational budgets, algorithms, or implementation.
- Examples
- Uses “Let’s think step-by-step,” detailed instructions, confidence requests, and zero-shot or few-shot prompts as examples.
- What you can do after
- Apply it via prompting by requesting stepwise reasoning, while recognizing that improvements vary across tasks and settings.
Read how Introduction to Generative AI, Second Edition explains inference-time compute scaling →
Related concepts
- language modeluses · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- multi-step reasoninguses · Multi-step reasoning derives a conclusion, solution, or decision through multiple connected intermediate inferential or problem-solving steps.
- tokenuses · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
- trajectoryuses · A trajectory is an ordered sequence of states, actions, events, or operations that records or describes progression through successive steps or over time.
- chain-of-thought promptinguses · Chain-of-thought prompting is a technique that elicits or guides a model to produce or follow intermediate reasoning steps before a final answer.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.