Reinforcement learning

As explained in Build a Reasoning Model (From Scratch)

A training approach where a model learns from rewards or feedback to improve its behavior.

Improves an LLM’s reasoning during training by rewarding responses that produce desirable outcomes. It is usually a post-training stage after pretraining and sometimes instruction fine-tuning, although reasoning-focused RL can also start from a pretrained base model. section 1.3 section 6.1

A practical RL workflow for reasoning models includes:

• Frame reasoning as sequences of actions and outcomes rather than explicit rule execution. section 1.5
• Sample multiple responses for each prompt and compute group-relative learning signals from their rewards. section 6.0
• Use task-correctness rewards in reinforcement learning with verifiable rewards (RLVR), or human feedback in RLHF. section 6.0
• Update model weights with group-based policy optimization to reinforce higher-reward reasoning behavior. section 6.0

RL differs from inference-time scaling: RL invests computation during training to improve later accuracy, while inference-time scaling spends more computation per generated answer. The two approaches can be combined. section 6.1

Figure 6.4 Road map of this chapter. After this section’s brief introduction to RL for LLMs, we’ll discuss the difference between two RL stages, RLHF and RLVR. In the rest of the chapter, we’ll focus on implementing RLVR using the GRPO algorithm.

Use the roadmap to place RL after the introductory distinction between RLHF and RLVR and before implementation with GRPO. section 6.1

How Build a Reasoning Model (From Scratch) covers it

How deep it goes

Explains reward signals, response groups, policy updates, and RLHF-versus-RLVR distinctions, then points toward GRPO implementation.

Examples

Uses reasoning LLMs, task-correctness rewards, and classic agents trained for chess, Go, or video games as examples.

What it assumes

Assumes a pretrained language model and optionally instruction fine-tuning, while introducing LLM-specific RL rather than general RL.

What you can do after

Apply RL to LLM reasoning by selecting reward signals, sampling response groups, and updating weights with group-based policy optimization.

How it connects

Connects reinforcement learning with pretraining, instruction fine-tuning, preference tuning, inference-time scaling, RLHF, RLVR, and policy optimization.

liveBookmanning.com

Read Build a Reasoning Model (From Scratch) on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackReinforcement learning is a machine learning paradigm in which an agent learns behavior by optimizing rewards obtained through interaction with an environment.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsReinforcement learning is a machine-learning paradigm in which an agent learns action policies from interactions and evaluative feedback.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksReinforcement learning is a machine-learning paradigm in which an agent learns behavior by interacting with an environment and receiving rewards or other evaluative feedback.
• 

  liveBookmanning.com
  Sutskever's ListMachine learning in which an agent learns behavior through rewards or penalties from interaction with an environment.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionReinforcement learning is a machine-learning technique in which an agent learns behavior through interaction with an environment and feedback such as rewards.
• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)Reinforcement learning is a machine-learning approach in which an agent learns a decision policy through interactions with an environment and feedback from rewards or penalties.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionReinforcement learning is a machine-learning paradigm in which an agent learns behavior through rewards for useful actions and penalties for harmful or unsuccessful actions in an environment.
• 

  liveBookmanning.com
  The AI Pocket BookA learning approach in which an agent improves by trying actions and using feedback from the results.
• 

  liveBookmanning.com
  Introduction to Generative AI, Second EditionReinforcement learning is machine learning in which an agent learns behavior from rewards or penalties produced by interactions with an environment.
• 

  liveBookmanning.com
  Grokking AI ApplicationsReinforcement learning is a machine learning approach in which an agent learns through trial and error while interacting with an environment to achieve goals.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA learning setup where an agent improves by interacting with an environment and receiving feedback, used here as a way to keep generating fresh training problems for LLMs.
• 

  liveBookmanning.com
  Grokking Machine LearningA machine-learning approach in which an agent learns to choose actions by interacting with an environment and receiving rewards or punishments.
• 

  liveBookmanning.com
  Acing the CCNA Exam: Volume 2 Advanced Networking and SecurityReinforcement learning is a machine-learning approach in which an agent learns behavior through interactions with an environment and feedback about its actions.
• 

  liveBookmanning.com
  Causal AIA machine-learning paradigm in which an agent learns actions or policies by interacting with an environment and receiving rewards or penalties.
• 

  liveBookmanning.com
  Inside AIReinforcement learning is a machine-learning technique in which an agent learns behavior by interacting with an environment and receiving feedback for its actions.
• 

  liveBookmanning.com
  Data Without LabelsA machine-learning paradigm in which an agent improves its behavior through sequential interactions, trial and error, and feedback from rewards or penalties.
• 

  liveBookmanning.com
  Data-Driven CybersecurityReinforcement learning is a machine-learning approach in which an agent learns decisions by interacting with an environment and optimizing cumulative feedback.

Compare how 18 books explain reinforcement learning →

Related concepts

• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.
• machine learningkind of · Machine learning is a field of computational methods in which systems learn patterns or relationships from data to make predictions, decisions, or other outputs.
• AI agentuses · An AI agent is a computational system that uses artificial intelligence to interpret information, select actions, and pursue goals through interaction with an environment.
• environmentuses · An environment is the external context, including its entities and conditions, in which a system, process, or agent exists or operates.
• reinforcement learning with human feedbackused by · Reinforcement learning with human feedback is a model-training approach that uses human evaluations or preferences to guide optimization of model behavior.
• fine-tuningcontrasts with · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.

This explainer was generated by Manning's knowledge graph from the text of Build a Reasoning Model (From Scratch). The section links point to the parts of the book it draws on.