Cover of Build a Reasoning Model (From Scratch)

What you will learn in Build a Reasoning Model (From Scratch)

by Sebastian Raschka

Read it on liveBook Get the book at manning.com

Chapter by chapter

Chapter 1: Understanding reasoning models

The chapter introduction motivates reasoning methods for complex LLM tasks and agents, and explains why the book uses checkable math examples while considering the costs of added reasoning machinery.

1.1 Defining reasoning in the context of LLMs · 1.2 Understanding the standard LLM training pipeline · 1.3 Improving LLM reasoning with training and inference techniques · 1.4 Pattern matching vs. logical reasoning · 1.5 Simulating reasoning without explicit rules · 1.6 Why build reasoning models from scratch? · 1.7 A road map to building reasoning models from scratch · 1.8 Summary

What you learn in chapter 1 →

Chapter 2: Generating text with a pretrained LLM

The chapter establishes why it starts with a pretrained base LLM before adding reasoning techniques in later chapters.

2.1 Introducing LLMs for text generation · 2.2 Setting up the coding environment · 2.3 Understanding hardware needs and recommendations · 2.4 Preparing input texts for LLMs · 2.5 Loading pretrained models · 2.6 Understanding the sequential LLM text generation process · 2.7 Coding a minimal text generation function · 2.8 Faster inference via KV caching · 2.9 Faster inference via PyTorch model compilation · 2.10 Summary

What you learn in chapter 2 →

Chapter 3: Evaluating reasoning models

The chapter introduction defines evaluation as quantitative testing against reference answers and identifies math-answer verification as the chapter’s approach.

3.1 Building a math verifier · 3.2 Loading a pretrained model to generate text · 3.3 Implementing a wrapper for easier text generation · 3.4 Extracting the final answer box · 3.5 Normalizing the extracted answer · 3.6 Verifying mathematical equivalence · 3.7 Grading answers · 3.8 Loading the evaluation dataset · 3.9 Evaluating the model · 3.10 Summary

What you learn in chapter 3 →

Chapter 4: Improving reasoning with inference-time scaling

The chapter introduction establishes that reasoning accuracy can improve during text generation without retraining or modifying the model.

4.1 Introduction to inference-time scaling · 4.2 Loading a pretrained model · 4.3 Generating better responses with chain-of-thought prompting · 4.4 Controlling output diversity with temperature scaling · 4.4.1 Understanding the process of selecting the next token · 4.4.2 Rescaling token scores (logits) via a temperature parameter · 4.4.3 Sampling the next token from a probability distribution · 4.4.4 Adding temperature scaling to the text generation function · 4.5 Balancing diversity and coherence with top-p sampling · 4.5.1 Selecting a subset of top-p tokens · 4.5.2 Adding a top-p filter to the text generation function · 4.6 Improving response accuracy with self-consistency · 4.7 Summary

What you learn in chapter 4 →

Chapter 5: Inference-time scaling via self-refinement

Introduces self-refinement as an inference-time scaling technique and contrasts it with self-consistency.

5.1 Scoring and iteratively improving model responses · 5.2 Loading a pretrained model · 5.3 Scoring LLM responses with a rule-based score · 5.4 Understanding token probability scores · 5.5 From token probability scores to log probabilities · 5.6 Scoring model confidence with log probabilities · 5.7 Self-refinement through iterative feedback · 5.8 Coding the self-refinement loop · 5.9 Summary

What you learn in chapter 5 →

Chapter 6: Training reasoning models with reinforcement learning

The introduction places reinforcement learning among the ways to improve reasoning models, contrasting training methods with increased inference compute.

6.1 Introduction to RL for LLMs · 6.1.1 The original RL pipeline with human feedback (RLHF) · 6.1.2 From human feedback to verifiable rewards · 6.2 RLVR using GRPO · 6.2.1 High-level GRPO intuition via a chef analogy · 6.2.2 The high-level GRPO procedure · 6.3 Loading a pretrained model · 6.4 Loading a MATH training subset · 6.5 Sampling rollouts · 6.6 Calculating rewards · 6.7 Preparing learning signals from rollouts via advantages · 6.8 Scoring rollouts with sequence log probabilities · 6.9 From advantages to policy updates via the GRPO loss · 6.10 Putting everything together in a single GRPO function · 6.11 Implementing the GRPO training loop · 6.12 Loading and evaluating saved model checkpoints · 6.13 Summary

What you learn in chapter 6 →

Chapter 7: Improving GRPO for reinforcement learning

The chapter picks up from an implemented GRPO baseline and identifies training instability as a concern during longer runs.

7.1 Improving GRPO · 7.2 Tracking GRPO performance metrics · 7.2.1 Executing a GRPO training run · 7.2.2 Inspecting the GRPO training run · 7.3 Tracking more advanced GRPO performance metrics · 7.3.1 Advantage tracking · 7.3.2 Entropy tracking · 7.3.3 Plotting additional GRPO metrics · 7.4 Stabilizing sequence-level GRPO using clipped policy ratios · 7.4.1 Computing clipped policy ratios · 7.4.2 Training with clipped policy ratios · 7.5 Controlling how much the model changes with a KL term · 7.5.1 Implementing the KL loss term · 7.5.2 Training with a KL loss term · 7.6 Adding an explicit format reward · 7.6.1 Using <think> tokens · 7.6.2 Training a model to emit <think> tokens · 7.6.3 More GRPO modifications, tips, and tricks · 7.7 Summary

What you learn in chapter 7 →

Chapter 8: Distilling reasoning models for efficient reasoning

The chapter introduces distillation as a training-time way to improve a smaller reasoning model using outputs from a larger model.

8.1 Introducing model distillation for reasoning tasks · 8.2 Generating a dataset for reasoning distillation · 8.3 Loading the MATH training dataset for distillation · 8.4 Building training examples · 8.4.1 Loading and understanding the tokenizer · 8.4.2 Formatting and tokenizing the dataset · 8.4.3 Filtering and splitting the dataset · 8.5 Loading a pretrained model · 8.6 Computing the training and validation losses · 8.7 Implementing the training loop for distillation · 8.8 Evaluating the distilled model · 8.9 Future directions for reasoning models · 8.10 Conclusions · 8.10.1 What’s next · 8.10.2 Staying up to date in a fast-moving field · 8.11 Summary

What you learn in chapter 8 →

Concepts explained in this book

Group Relative Policy OptimizationA reinforcement-learning method that uses groups of rollouts for the same prompt and derives learning signals from their relative performance.
log probabilityThe logarithm of a model-assigned probability, used here to score how likely a generated token or whole sequence is under the current parameters.
generationThe process of producing new textual output from a language model given prior context or instructions.
temperatureA token sampling method that draws from a probability distribution after temperature rescaling to control randomness.
distillationA model adaptation approach that trains a smaller student model to imitate a larger teacher model’s outputs or behavior.
inference-time compute scalingImproving a trained model's output quality by spending more computation at inference time without changing its weights.
self-consistencyAn inference technique that generates multiple candidate responses and selects or combines them to improve answer quality.
token probabilityThe probability assigned by the model to a candidate token at a particular position in the output sequence.
softmaxA normalization function that converts logits into a probability distribution over possible next tokens.
language modelA neural language model trained on large text corpora to predict and generate text.
multi-step reasoningThe process of deriving conclusions from premises through explicit inference rather than surface pattern recognition.
Kullback–Leibler divergenceA measure of how different two probability distributions are, used here for soft distillation.
continuous improvementAn inference-time procedure that iteratively revises a model answer and keeps a revision only if it scores better than the current answer.
top-p filteringA decoding method that restricts sampling to the smallest set of tokens whose cumulative probability reaches a chosen threshold p.
reasoning modelA language model optimized to deliberate over complex problems, often producing intermediate steps or using tools to improve answer quality.
tokenizerA component that splits text into token IDs for model input, with this section focusing on the Qwen3 reasoning variant that supports reasoning tags.
reinforcement learning with human feedbackA training approach that uses human preference labels to optimize a language model toward responses humans prefer.
confidenceThe model-assigned probability used here as a scorer for how strongly an LLM prefers a candidate answer.
joint probability distributionThe probability of an entire token sequence under a model, computed by multiplying conditional token probabilities across positions.
torch.prodPyTorch's product operation used here to multiply per-token probabilities into a sequence score.
reinforcement learning with verifiable rewardsA reinforcement learning approach that uses automatically checkable correctness rewards instead of human preference signals.
reinforcement learningA training approach where a model learns from rewards or feedback to improve its behavior.
clipped policy ratioA policy-optimization constraint that limits how far a new policy’s probability ratio can move from an old policy’s ratio for the same sampled response.
pretrainingThe initial training stage in which a language model learns general statistical structure from large amounts of data before task-specific adaptation.
templateA fixed prompt structure used to shape how an LLM formats and reasons in its response.
answer checking pipelineA deterministic workflow that extracts a model's answer and compares it to the reference for correctness.
logprob scorerA scoring method that ranks answers by their average log probability under the model, favoring fluent and likely text.
trainingA method that improves model behavior by further updating the model through training.
post-trainingMethods applied after initial pretraining to improve model behavior or capabilities.
sequence_logprobA sequence-scoring function that sums token log probabilities to obtain a whole-response log probability.
evaluationThe process of assessing model performance on held-out data or benchmarks.
probability distributionA normalized set of values over tokens that sum to 1 and can be sampled from to choose the next token.
response-formatting rewardA reward component that gives the model credit for producing outputs in the required tag format, regardless of whether the answer is correct.
trajectoryA generated response trajectory sampled from the model, treated as one complete candidate output in policy optimization.
torch.log_softmaxA PyTorch function that computes log-softmax over logits to produce log probabilities in a numerically stable way.
<think> tagA special markup tag used to enclose hidden reasoning steps in model output.
inference modeA runtime setting that disables gradient tracking and other training-time behavior for efficient model execution.
preference tuningA post-training method that adjusts a model to produce responses aligned with preferred outputs or styles.
student modelA smaller model trained to imitate a teacher model's reasoning traces and answers during distillation.
pattern recognitionThe process of completing text or answers by relying on learned correlations and recurring examples instead of formal reasoning.
reasoning methodologyA method applied to a language model to improve explicit step-by-step reasoning behavior.
torch.compile()An optimization process that transforms a model's computation graph into more efficient executable code for faster inference.
teacher modelA larger model whose outputs or signals are used as training targets for a smaller student model during distillation.
advantage functionA normalized per-rollout signal computed from rewards that indicates how much better or worse a rollout was than the group average.
verifierA component that checks whether a candidate answer is correct or acceptable, often by comparing it against a reference.
computation graphA recorded graph of differentiable operations in PyTorch that enables gradient computation during backpropagation.
verifiable rewardA reward signal that can be automatically checked against a correct answer or objective criterion.
cross entropy lossA loss function that measures the average negative log probability assigned to the correct target tokens.
numerical stabilityThe property of a computation remaining accurate and avoiding underflow or rounding errors when values are very small or very large.
rule-based scoring functionA manually specified scoring method that assigns numeric values to model outputs using explicit rules rather than learned parameters.
answer boxA final answer presented inside boxed notation so it can be extracted more reliably from the response text.
regular expressionA pattern-matching technique for searching and extracting text using formal expression syntax.
end of text tokenA special token used to mark the end of a document or generated sequence and stop generation.
stopping criterionA criterion that terminates generation when met, such as encountering an end-of-sequence token.
policy gradient methodThe reinforcement-learning objective used to adjust a policy so actions with better outcomes become more likely and worse ones less likely.
symbolic AIReasoning based on explicit symbolic rules rather than statistical associations learned from data.
MATH-500A curated benchmark dataset of 500 math problems sampled from the original MATH dataset for evaluating reasoning models.
answer-only lossA training loss computed only on answer tokens and not on prompt tokens.
equality_checkA function that decides whether two mathematical expressions are equivalent after normalization.
verification pipelineThe ordered process used to evaluate model answers, from loading the model through scoring on a dataset.
SymPyAn open-source Python library for symbolic mathematics used here to parse and compare expressions for equivalence.
tokenizationThe process of rendering text into a model-ready format and splitting it into tokens/IDs for training.
entropyA measure of uncertainty in the model's output distribution during generation, where low entropy indicates collapse and high entropy indicates randomness.
tokenA discrete text unit used as the basic element of language-model generation.
cumulative sumThe running total of sorted token probabilities used here to determine which tokens remain under a top-p threshold.
pipPython’s package installer used to install packages from a terminal.
PyTorchA deep learning library used here to represent tensors and define optimizers that minimize losses.
requirements.txtA file listing Python package dependencies to install for a project.
uvA Python package and project manager that can create environments and resolve dependencies automatically.
mathematical equivalenceThe property that two expressions represent the same mathematical value despite differing surface forms.
greedy algorithmA decoding strategy that always selects the highest-probability next token instead of sampling randomly.
equalityThe process of determining whether two mathematical expressions represent the same value or form symbolically.
evaluation pipelineAn ordered process that loads a model, generates outputs, and scores them against a dataset or reference answers.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.