liveBook · Manning

Chapter 11: Teach LLMs to reason with Verifiable Feedback using GRPO

What you will learn in chapter 11 of Applied Reinforcement Learning

This introduction explains when a programmatic verifier can replace a learned reward model for language-model tasks.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction explains when a programmatic verifier can replace a learned reward model for language-model tasks.

Key terms: ground-truth answer, verifier, reward model, human preference pairs, Reinforcement Learning with Verifiable Rewards

11.1 Teaching language models to reason on financial reports

This section frames financial question answering with FinQA as a language-model reinforcement learning task with objectively verifiable, sparse rewards.

Key terms: verifiable rewards, gold answer, relative tolerance, RLVR, FinQA, pre-text, table, post-text, exe_ans, state

11.2 Reinforcement Learning with Verifiable Rewards

This section explains verifiable rewards and how a FinQA verifier scores and normalizes numerical answers.

Key terms: Reinforcement Learning with Verifiable Rewards, RLVR, deterministic verifier, learned reward model, gold answer, relative tolerance, format bonus, reward_correct, reward_format, boxed answer

11.3 LoRA: fine-tuning large models with tiny adapters

This section explains how LoRA reduces language-model fine-tuning memory by training small adapters while preserving access to the frozen base model.

Key terms: optimizer state, activations, backward pass, bfloat16, LoRA, low-rank subspace, rank, frozen weight, adapter, scaling constant

11.4 GRPO: group relative policy optimization

This section explains how GRPO uses relative rewards from groups of responses to update a language model and discusses the conditions needed for useful, stable training.

Key terms: GRPO, value head, group-relative baseline, group-relative advantage, RLVR verifier, group mean, group standard deviation, zero-gradient problem, temperature, group diversity

11.5 Training stability in GRPO

This section introduces memory engineering and training dynamics as practical challenges in GRPO training.

11.5.1 Memory for logit computation

This section explains why vocabulary-wide logits consume substantial GRPO training memory and describes two ways to reduce peak memory use.

Key terms: logits, softmax, logit tensor, backward pass, log_softmax, logsumexp, log-probability, float32, micro-batches, group

11.5.2 Computing the reference policy without a second copy

This section explains how to compute reference and policy log-probabilities without a second model copy by toggling LoRA adapters.

Key terms: reference policy, base model, adapters, reference log-probabilities, policy log-probabilities, no_grad()

11.5.3 Gradient checkpointing

This section explains how gradient checkpointing reduces activation memory during LoRA training and distinguishes runnable code from useful RL fine-tuning.

Key terms: LoRA, forward activations, backpropagation, gradient checkpointing, RL fine-tuning, training stability

11.5.4 Diagnosing training stability with KL divergence

This section explains how to diagnose GRPO training stability by tracking KL divergence from a frozen reference model.

Key terms: KL divergence, current policy, frozen reference model, KL penalty

11.6 Five lessons from training GRPO at scale

11.6.1 Lesson 1: Learning rate

This section uses training results to explain learning-rate selection and KL-based diagnosis for LoRA with GRPO.

Key terms: LoRA, learning rate, GRPO, full fine-tuning, KL

11.6.2 Lesson 2: Too hot RL vs too cold RL

This section explains how overly aggressive RL training can cause policy drift without improving accuracy.

Key terms: over-optimization, KL penalty, reference, KL, boxed answer format, accuracy

11.6.3 Lesson 3: KL penalty

This section explains how a KL penalty can stabilize training, while noting that some models can train successfully without one.

Key terms: KL penalty, KL, klcoef, anchor

11.6.4 Lesson 4: Base model impact

This section explains how a base model’s starting performance affects the measured gains available from GRPO fine-tuning.

Key terms: base model, headroom, GRPO, RLVR

11.6.5 Lesson 5: KL signal is the key

This section explains how stable KL divergence and stalled accuracy signal GRPO training convergence.

Key terms: KL, sampled accuracy, convergence, equilibrium

11.7 Building the financial reasoning agent

This section introduces a financial reasoning pipeline combining RLVR, LoRA, and GRPO in a hand-written implementation.

Key terms: RLVR, deterministic reward, LoRA, GRPO, value head

11.7.1 Pipeline configuration

This section organizes the pipeline’s tunable settings into dataclasses for the model, LoRA injection, and GRPO training.

Key terms: dataclass, configuration classes, LoRA injection, GRPO training

11.7.2 LoRA injection

This section describes injecting LoRA adapters into pretrained linear layers and the resulting reduction in trainable parameters.

Key terms: LoRALinear, nn.Linear, low-rank matrices A and B, frozen base weight, trainable adapter, inject_lora, trainable parameters, gradients, Adam optimizer state, rank 32

11.7.3 FinQA data pipeline and prompt formatting

This section describes how FinQA records become Qwen-formatted prompts paired with gold answers for training.

Key terms: FinQA, pre_text, table, post_text, qa, question

11.7.4 RLVR verifier: extracting and scoring answers

This section explains how the RLVR verifier extracts and scores financial answers while allowing for percentage representation differences.

Key terms: verifier, reward function, extract_answer, compute_reward, gold answer, scalar reward, numbers_match, percentage, decimal fraction

11.7.5 GRPO training: rollouts, advantages, and the update step

This section outlines the GRPO training step and identifies the memory-efficient log-probability function used by the trainer.

Key terms: grpo_step, rollouts, RLVR rewards, group-relative advantages, clipped PPO surrogate objective, KL diagnostic, _token_log_probs, per-token log-probabilities, full softmax tensor

11.7.6 Evaluation: comparing base and fine-tuned models

This section compares base and GRPO-fine-tuned models on FinQA, interprets their training curves, and explains the gains and limits of verifiable-reward training.

Key terms: LoRA adapters, base model, fine-tuned model, greedy decoding, FinQA test set, format bonus, boxed-answer format, verifier, format rate, mean reward

11.8 Policy-based methods beyond language models

The section establishes that policy-based reinforcement learning primitives apply beyond language models to agents acting in reward-giving environments.

Key terms: policy gradient updates, trust-region, KL constraints, group-relative advantage estimation, baseline advantage estimation

11.8.1 Robotics and physical control

This section connects policy-gradient methods in language-model fine-tuning to robotic control, contrasting their inputs and outputs while emphasizing stable policy updates.

Key terms: PPO, continuous control, state space, action space, policy gradient, trust-region constraint, KL, policy

11.8.2 Recommendation systems at scale

This section frames recommendations as a policy problem and connects exploration in large-scale recommendation systems to sampling diversity in GRPO.

Key terms: policy, state, action, reward, engagement, exploration versus exploitation, sampling, diversity, gradient, temperature

11.8.3 Drug discovery and molecular design

This section applies sequence-level reinforcement learning and verifiable rewards to molecular design.

Key terms: sequence-level RL, chemical grammar, binding affinity, toxicity, pharmacokinetics, verifier, molecular docking, binding affinity score, verifiable rewards, wet-lab assay

11.8.4 Code generation with execution feedback

This section explains how code execution supplies verifiable feedback for policy-gradient training of code generation models.

Key terms: test-suite execution, reward signal, group-relative advantages, verifier function, format reward, compiles, fails to parse

11.8.5 Operations research and combinatorial optimization

This section describes how policy-based reinforcement learning can learn heuristics for large-scale combinatorial optimization and why those heuristics can benefit businesses.

Key terms: combinatorial problems, learned heuristics, attention mechanisms, sequential construction, PPO, REINFORCE, route length, learned policy, traffic conditions, demand shocks

11.8.6 Clinical decision support and treatment sequencing

This section illustrates policy-based reinforcement learning in clinical treatment and explains how shared optimization tools transfer across domains through domain-specific rewards and verification.

Key terms: sequential, retrospective patient records, policies, safety constraint, KL penalty, standard of care, policy-based methods, baseline, verifier, reward design

11.9 Summary

Concepts explained in this chapter

← Chapter 10: Fine-tuning large language models with PPO Chapter 12: Building Reinforcement Learning systems: LLM-guided RL agent for portfolio management →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.