Chapter 11: Teach LLMs to reason with Verifiable Feedback using GRPO
This introduction explains when a programmatic verifier can replace a learned reward model for language-model tasks.
After this chapter you will be able to
- Verifiable answers and rewards. Explain why a task with a mechanically checkable ground-truth answer can use a verifier instead of a learned reward model.
- Verifiable Financial Reasoning. Explain why financial question answering suits reinforcement learning with verifiable rewards, and contrast its feedback with the medical chatbot's preference-based judgments.
- FinQA Example and Reasoning. Describe the contents of a FinQA example and explain why answering it takes more than arithmetic alone.
- FinQA MDP Mapping. Trace how a FinQA response maps to states, actions, transitions, an episode, and a final reward.
- Sparse Reward and Answer Format. Explain why the final answer's format matters when FinQA feedback is sparse.
- Verifiable rewards and learned models. Explain when a deterministic verifier can replace a learned reward model and why that simplifies training.
- FinQA reward signal structure. Trace how the FinQA verifier combines numerical correctness and the boxed-answer format bonus, including why tolerance is used.
- Low-rank adapter mechanism. Explain how LoRA represents a low-rank weight update and why this reduces trainable parameters and optimizer memory.
- Adapter-free reference policy. Explain how disabling LoRA adapters supplies a frozen reference policy without a second model copy.
- Group Relative Baseline. Explain how GRPO replaces PPO’s value head with a baseline derived from responses to the same prompt, and why this simplifies training.
- Normalized Response Advantages. Trace how GRPO samples and scores responses to one prompt, then turns their rewards into relative advantages.
- Zero Gradient Conditions. Explain why uniform rewards give no useful GRPO signal and how sampling diversity and initial model capability affect this problem.
- Token Objective and Length. Describe how a response-level advantage is used in the token-level clipped objective and why each response is normalized by its length.
- KL Reference and Stability. Explain how the KL penalty is included in the GRPO objective and why the section’s experiments retained it.
- Logit memory bottleneck. Explain why logits create a large memory burden during GRPO training.
- Memory-efficient log-probability computation. Describe how the section reduces memory use when computing token log probabilities.
- Gradient-accumulating group micro-batches. Explain how group micro-batching reduces peak memory without changing the objective.
- Shared reference policy computation. Explain how the same model infrastructure computes reference and policy log-probabilities without a second copy, and why these passes hold no activation memory.
- Gradient checkpointing memory trade-off. Explain why LoRA training still needs frozen-layer activations and how gradient checkpointing trades computation time for memory.
- KL Divergence Training Diagnostics. Interpret KL divergence between the current policy and the frozen reference model as a sign of stalled, unstable, or productive training.
- LoRA learning rate scale. Explain why the section recommends a larger learning rate for LoRA with GRPO than for full-model RLHF, using the reported training results.
- KL movement as diagnosis. Describe how the section uses KL movement to diagnose a GRPO run that shows no improvement.
- Over optimization warning signs. Explain how overly aggressive training affected the models and how to recognize over-optimization from KL divergence and accuracy.
- KL penalty and stability. Explain how a KL penalty can stabilize training and why its necessity depends on the model.
- Base model improvement headroom. Explain why a stronger starting model can show a smaller measured gain from GRPO, using the FinQA comparison.
- KL convergence signal. Explain how KL divergence and sampled accuracy indicate that training has converged for a given configuration, and what that implies for continued training.
- Three Pipeline Component Roles. Describe the roles of RLVR, LoRA, and GRPO in the financial reasoning pipeline.
- Pipeline configuration dataclasses. Describe how the pipeline organizes its tunable settings and what the three configuration classes govern.
- LoRA linear layer wrapping. Explain how LoRALinear adapts an existing linear layer while keeping its base weight frozen.
- Adapter training parameter savings. Describe which parameters receive gradients, optimizer state, and updates after LoRA injection, and illustrate the savings with the section's model example.
- FinQA prompt and answer preparation. Describe how a FinQA record is converted into a training example for Qwen.
- Verifier reward function. Describe how the verifier turns a model's generated text and a gold answer into a reward.
- Percentage representation matching. Explain why numbers_match accepts differently scaled predictions and illustrate its behavior for a gold answer of 3.8.
- GRPO training step sequence. Trace the main stages of grpo_step, from a batch of prompts through the PPO-style update.
- Held-out evaluation and results. Describe how the base and fine-tuned models were compared and summarize the accuracy improvement.
- Format rewards and parsing. Explain why a small format bonus produced a large change and why that change mattered for verification.
- Improved strategies and limits. Explain how GRPO improved the model's use of existing knowledge, using the financial-report example, and state its limits.
- Base model improvement headroom. Explain why a stronger base model may gain less from the same GRPO recipe.
- Domain agnostic policy methods. Explain why the policy-based ideas introduced for language models also apply in other domains.
- Robotic Control Policy Mapping. Describe how policy-gradient methods apply to robotic control and how the state and action spaces differ from those in language-model fine-tuning.
- Policy Update Stability. Explain why constraining policy changes matters in both robotic control and language-model fine-tuning.
- Recommendation Policy Components. Describe the state, action, and reward in a recommendation policy over user sessions.
- Exploration and Sampling Diversity. Explain why recommendation policies need exploration and how the section connects this need to sampling diversity in GRPO.
- Molecular sequence generation. Explain why molecular generation fits sequence-level reinforcement learning and what properties it can target.
- Docking verifier and bottleneck. Describe how molecular docking provides verifiable feedback and why the verifier is the bottleneck in this application.
- Execution Feedback Training Pipeline. Trace how test-suite results are used to fine-tune a code generation model and describe how this adapts the FinQA pipeline.
- Sequential construction with learned policies. Explain how learned policies approach combinatorial optimization, using vehicle routing as an example.
- Clinical Safety Constraint. Explain why a clinical RL policy should remain close to the standard of care and how this resembles the KL penalty in LLM fine-tuning.
- Transferable Optimization and Verification. Explain how policy-based optimization tools can transfer between domains and why a trustworthy domain-specific verifier matters.
What it builds on
- Learning a Reward Model from Human Preferences (chapter 10) · The learned reward model based on human preferences that a verifier can replace.
- Mapping Language Model to a Markov Decision Process (chapter 10) · Language generation can be mapped to an MDP in which tokens are actions.
- Proximal Policy Optimization: From REINFORCE to PPO (chapter 10) · The PPO context in which a reference policy and KL penalty help control policy drift.
- Tradeoff between exploitation and exploration (chapter 5) · The tradeoff between exploitation and exploration.
- The REINFORCE algorithm (chapter 9) · REINFORCE as a policy-based reinforcement learning algorithm
- Attention mechanism: learning what to look at (chapter 9) · attention mechanisms
Section by section
Introduction
This introduction explains when a programmatic verifier can replace a learned reward model for language-model tasks.
- Verifiable answers and rewards: Explain why a task with a mechanically checkable ground-truth answer can use a verifier instead of a learned reward model.
Why it matters: A checkable answer lets you use direct correctness feedback rather than learn a proxy for human preferences.
Key terms: ground-truth answer, verifier, reward model, human preference pairs, Reinforcement Learning with Verifiable Rewards
11.1 Teaching language models to reason on financial reports
This section frames financial question answering with FinQA as a language-model reinforcement learning task with objectively verifiable, sparse rewards.
- Verifiable Financial Reasoning: Explain why financial question answering suits reinforcement learning with verifiable rewards, and contrast its feedback with the medical chatbot's preference-based judgments.
Why it matters: A checkable final answer provides feedback without requiring human raters for every response. - FinQA Example and Reasoning: Describe the contents of a FinQA example and explain why answering it takes more than arithmetic alone.
Why it matters: FinQA tests whether a model can connect a financial question to the right evidence and calculation. - FinQA MDP Mapping: Trace how a FinQA response maps to states, actions, transitions, an episode, and a final reward.
Why it matters: This mapping turns generating a financial answer into a sequential reinforcement learning problem. - Sparse Reward and Answer Format: Explain why the final answer's format matters when FinQA feedback is sparse.
Why it matters: A correct calculation can fail to earn feedback if the verifier cannot extract its answer.
Key terms: verifiable rewards, gold answer, relative tolerance, RLVR, FinQA, pre-text, table, post-text, exe_ans, state
11.2 Reinforcement Learning with Verifiable Rewards
This section explains verifiable rewards and how a FinQA verifier scores and normalizes numerical answers.
- Verifiable rewards and learned models: Explain when a deterministic verifier can replace a learned reward model and why that simplifies training.
Why it matters: This distinction explains why numerical reasoning tasks can use direct feedback instead of a learned preference model. - FinQA reward signal structure: Trace how the FinQA verifier combines numerical correctness and the boxed-answer format bonus, including why tolerance is used.
Why it matters: Understanding the two independent reward components helps you interpret the feedback the model receives. - Financial answer normalization rules: Describe how the verifier handles percentage scales and financial notation before judging a numerical answer.
Why it matters: These rules prevent differences in financial notation from unnecessarily losing reward.
Key terms: Reinforcement Learning with Verifiable Rewards, RLVR, deterministic verifier, learned reward model, gold answer, relative tolerance, format bonus, reward_correct, reward_format, boxed answer
11.3 LoRA: fine-tuning large models with tiny adapters
This section explains how LoRA reduces language-model fine-tuning memory by training small adapters while preserving access to the frozen base model.
- Fine-tuning memory costs: Explain why full fine-tuning requires much more GPU memory than storing model weights alone.
Why it matters: Understanding the extra training memory explains why a model that fits on a GPU may not be practical to fully fine-tune there. - Low-rank adapter mechanism: Explain how LoRA represents a low-rank weight update and why this reduces trainable parameters and optimizer memory.
Why it matters: This is the central reason LoRA can fine-tune a large model with much less trainable-parameter memory. - Stable adapter initialization: Describe how LoRA initializes its two matrices and why those choices provide a stable starting point.
Why it matters: The initialization lets training begin from known base-model behavior without an unstable adapter gradient path. - Adapter-free reference policy: Explain how disabling LoRA adapters supplies a frozen reference policy without a second model copy.
Why it matters: Reusing the frozen base as a reference saves GPU memory while retaining a baseline against which to limit policy drift. - Activation memory and checkpointing: Explain why LoRA still needs activation memory and how gradient checkpointing trades computation for memory.
Why it matters: LoRA reduces optimizer memory but does not by itself eliminate the activation-memory bottleneck.
Key terms: optimizer state, activations, backward pass, bfloat16, LoRA, low-rank subspace, rank, frozen weight, adapter, scaling constant
11.4 GRPO: group relative policy optimization
This section explains how GRPO uses relative rewards from groups of responses to update a language model and discusses the conditions needed for useful, stable training.
- Group Relative Baseline: Explain how GRPO replaces PPO’s value head with a baseline derived from responses to the same prompt, and why this simplifies training.
Why it matters: The group baseline is the defining change that makes GRPO different from value-head-based PPO. - Normalized Response Advantages: Trace how GRPO samples and scores responses to one prompt, then turns their rewards into relative advantages.
Why it matters: Relative advantages translate verifier scores into a graded learning signal for the responses. - Zero Gradient Conditions: Explain why uniform rewards give no useful GRPO signal and how sampling diversity and initial model capability affect this problem.
Why it matters: GRPO cannot learn a preference from a group whose answers all receive the same score. - Token Objective and Length: Describe how a response-level advantage is used in the token-level clipped objective and why each response is normalized by its length.
Why it matters: These choices determine how a single answer’s reward affects its tokens without automatically favoring verbosity. - KL Reference and Stability: Explain how the KL penalty is included in the GRPO objective and why the section’s experiments retained it.
Why it matters: The reference penalty is a practical safeguard against the model drifting into a worse output distribution. - LoRA Parameter Updates: Distinguish the parameters updated by full fine-tuning from those updated by the section’s LoRA-based GRPO approach.
Why it matters: Updating only adapters makes the GRPO training approach more memory-efficient.
Key terms: GRPO, value head, group-relative baseline, group-relative advantage, RLVR verifier, group mean, group standard deviation, zero-gradient problem, temperature, group diversity
11.5 Training stability in GRPO
This section introduces memory engineering and training dynamics as practical challenges in GRPO training.
11.5.1 Memory for logit computation
This section explains why vocabulary-wide logits consume substantial GRPO training memory and describes two ways to reduce peak memory use.
- Logit memory bottleneck: Explain why logits create a large memory burden during GRPO training.
Why it matters: This explains why training memory grows so quickly with batch size, sequence length, and vocabulary size. - Memory-efficient log-probability computation: Describe how the section reduces memory use when computing token log probabilities.
Why it matters: Avoiding an additional vocabulary-sized tensor makes log-probability computation more memory-efficient. - Gradient-accumulating group micro-batches: Explain how group micro-batching reduces peak memory without changing the objective.
Why it matters: Micro-batching lets the same group objective be trained with a lower peak memory requirement.
Key terms: logits, softmax, logit tensor, backward pass, log_softmax, logsumexp, log-probability, float32, micro-batches, group
11.5.2 Computing the reference policy without a second copy
This section explains how to compute reference and policy log-probabilities without a second model copy by toggling LoRA adapters.
- Shared reference policy computation: Explain how the same model infrastructure computes reference and policy log-probabilities without a second copy, and why these passes hold no activation memory.
Why it matters: This avoids keeping a separate reference model and holding activation memory for these passes.
Key terms: reference policy, base model, adapters, reference log-probabilities, policy log-probabilities, no_grad()
11.5.3 Gradient checkpointing
This section explains how gradient checkpointing reduces activation memory during LoRA training and distinguishes runnable code from useful RL fine-tuning.
- Gradient checkpointing memory trade-off: Explain why LoRA training still needs frozen-layer activations and how gradient checkpointing trades computation time for memory.
Why it matters: This trade-off makes training feasible on a single GPU when activations would otherwise consume too much memory. - Runnable versus useful training: Distinguish making RL fine-tuning run from making it useful, including the two failures a badly chosen setting can cause.
Why it matters: A training run that completes can still fail to improve the model or even make it worse.
Key terms: LoRA, forward activations, backpropagation, gradient checkpointing, RL fine-tuning, training stability
11.5.4 Diagnosing training stability with KL divergence
This section explains how to diagnose GRPO training stability by tracking KL divergence from a frozen reference model.
- KL Divergence Training Diagnostics: Interpret KL divergence between the current policy and the frozen reference model as a sign of stalled, unstable, or productive training.
Why it matters: This diagnostic helps you distinguish a model that is learning productively from one that is frozen or drifting too far.
Key terms: KL divergence, current policy, frozen reference model, KL penalty
11.6 Five lessons from training GRPO at scale
11.6.1 Lesson 1: Learning rate
This section uses training results to explain learning-rate selection and KL-based diagnosis for LoRA with GRPO.
- LoRA learning rate scale: Explain why the section recommends a larger learning rate for LoRA with GRPO than for full-model RLHF, using the reported training results.
Why it matters: A learning rate suited to full-model fine-tuning can leave LoRA training making too little progress. - KL movement as diagnosis: Describe how the section uses KL movement to diagnose a GRPO run that shows no improvement.
Why it matters: Checking KL can reveal a lack of learning before you attribute poor results to the model or method.
Key terms: LoRA, learning rate, GRPO, full fine-tuning, KL
11.6.2 Lesson 2: Too hot RL vs too cold RL
This section explains how overly aggressive RL training can cause policy drift without improving accuracy.
- Over optimization warning signs: Explain how overly aggressive training affected the models and how to recognize over-optimization from KL divergence and accuracy.
Why it matters: Recognizing this pattern helps distinguish a model that is changing from one that is actually improving.
Key terms: over-optimization, KL penalty, reference, KL, boxed answer format, accuracy
11.6.3 Lesson 3: KL penalty
This section explains how a KL penalty can stabilize training, while noting that some models can train successfully without one.
- KL penalty and stability: Explain how a KL penalty can stabilize training and why its necessity depends on the model.
Why it matters: The need for a KL anchor can vary with the model, affecting whether training improves steadily or collapses.
Key terms: KL penalty, KL, klcoef, anchor
11.6.4 Lesson 4: Base model impact
This section explains how a base model’s starting performance affects the measured gains available from GRPO fine-tuning.
- Base model improvement headroom: Explain why a stronger starting model can show a smaller measured gain from GRPO, using the FinQA comparison.
Why it matters: Starting performance helps explain why the same fine-tuning recipe can produce different measured gains.
Key terms: base model, headroom, GRPO, RLVR
11.6.5 Lesson 5: KL signal is the key
This section explains how stable KL divergence and stalled accuracy signal GRPO training convergence.
- KL convergence signal: Explain how KL divergence and sampled accuracy indicate that training has converged for a given configuration, and what that implies for continued training.
Why it matters: This signal helps you decide when further training is unlikely to justify its compute cost.
Key terms: KL, sampled accuracy, convergence, equilibrium
11.7 Building the financial reasoning agent
This section introduces a financial reasoning pipeline combining RLVR, LoRA, and GRPO in a hand-written implementation.
- Three Pipeline Component Roles: Describe the roles of RLVR, LoRA, and GRPO in the financial reasoning pipeline.
Why it matters: These three roles explain how the pipeline combines feedback, affordable fine-tuning, and policy training.
Key terms: RLVR, deterministic reward, LoRA, GRPO, value head
11.7.1 Pipeline configuration
This section organizes the pipeline’s tunable settings into dataclasses for the model, LoRA injection, and GRPO training.
- Pipeline configuration dataclasses: Describe how the pipeline organizes its tunable settings and what the three configuration classes govern.
Why it matters: This organization makes it clear where the pipeline’s adjustable settings are defined.
Key terms: dataclass, configuration classes, LoRA injection, GRPO training
11.7.2 LoRA injection
This section describes injecting LoRA adapters into pretrained linear layers and the resulting reduction in trainable parameters.
- LoRA linear layer wrapping: Explain how LoRALinear adapts an existing linear layer while keeping its base weight frozen.
Why it matters: This is the layer-level mechanism that makes the fine-tuning parameter-efficient. - Adapter training parameter savings: Describe which parameters receive gradients, optimizer state, and updates after LoRA injection, and illustrate the savings with the section's model example.
Why it matters: Training only a small fraction of the model limits the parameters that need gradients and optimizer state.
Key terms: LoRALinear, nn.Linear, low-rank matrices A and B, frozen base weight, trainable adapter, inject_lora, trainable parameters, gradients, Adam optimizer state, rank 32
11.7.3 FinQA data pipeline and prompt formatting
This section describes how FinQA records become Qwen-formatted prompts paired with gold answers for training.
- FinQA prompt and answer preparation: Describe how a FinQA record is converted into a training example for Qwen.
Why it matters: This preparation gives the model a usable financial question and provides an answer against which its response can be checked.
Key terms: FinQA, pre_text, table, post_text, qa, question
11.7.4 RLVR verifier: extracting and scoring answers
This section explains how the RLVR verifier extracts and scores financial answers while allowing for percentage representation differences.
- Verifier reward function: Describe how the verifier turns a model's generated text and a gold answer into a reward.
Why it matters: The verifier turns an answer into a reward signal for reinforcement learning. - Percentage representation matching: Explain why numbers_match accepts differently scaled predictions and illustrate its behavior for a gold answer of 3.8.
Why it matters: A generous numerical match avoids penalizing an answer solely for how a ratio is represented.
Key terms: verifier, reward function, extract_answer, compute_reward, gold answer, scalar reward, numbers_match, percentage, decimal fraction
11.7.5 GRPO training: rollouts, advantages, and the update step
This section outlines the GRPO training step and identifies the memory-efficient log-probability function used by the trainer.
- GRPO training step sequence: Trace the main stages of grpo_step, from a batch of prompts through the PPO-style update.
Why it matters: This sequence shows how verified response rewards feed into the model’s training update. - Memory-efficient token log-probabilities: Describe what the trainer’s memory-efficient token log-probability function avoids.
Why it matters: Avoiding the full softmax tensor is one of the memory-saving measures in the training loop.
Key terms: grpo_step, rollouts, RLVR rewards, group-relative advantages, clipped PPO surrogate objective, KL diagnostic, _token_log_probs, per-token log-probabilities, full softmax tensor
11.7.6 Evaluation: comparing base and fine-tuned models
This section compares base and GRPO-fine-tuned models on FinQA, interprets their training curves, and explains the gains and limits of verifiable-reward training.
- Held-out evaluation and results: Describe how the base and fine-tuned models were compared and summarize the accuracy improvement.
Why it matters: This comparison shows what the reinforcement learning step added to the same underlying model. - Format rewards and parsing: Explain why a small format bonus produced a large change and why that change mattered for verification.
Why it matters: A small, well-designed reward can make answers usable by the verifier without changing the correctness objective. - Training curves and decoding: Describe the main training-curve trends and distinguish sampled training accuracy from final greedy evaluation accuracy.
Why it matters: The training and final accuracy numbers must be interpreted in light of their different decoding methods. - Improved strategies and limits: Explain how GRPO improved the model's use of existing knowledge, using the financial-report example, and state its limits.
Why it matters: The example shows both the practical value of coaching a capable model and why that coaching has limits. - Base model improvement headroom: Explain why a stronger base model may gain less from the same GRPO recipe.
Why it matters: Starting performance affects how much improvement the same training recipe can deliver.
Key terms: LoRA adapters, base model, fine-tuned model, greedy decoding, FinQA test set, format bonus, boxed-answer format, verifier, format rate, mean reward
11.8 Policy-based methods beyond language models
The section establishes that policy-based reinforcement learning primitives apply beyond language models to agents acting in reward-giving environments.
- Domain agnostic policy methods: Explain why the policy-based ideas introduced for language models also apply in other domains.
Why it matters: The methods used to train language models belong to a broader family of reinforcement learning ideas.
Key terms: policy gradient updates, trust-region, KL constraints, group-relative advantage estimation, baseline advantage estimation
11.8.1 Robotics and physical control
This section connects policy-gradient methods in language-model fine-tuning to robotic control, contrasting their inputs and outputs while emphasizing stable policy updates.
- Robotic Control Policy Mapping: Describe how policy-gradient methods apply to robotic control and how the state and action spaces differ from those in language-model fine-tuning.
Why it matters: The same policy-gradient ideas can transfer from generating language to controlling physical movement. - Policy Update Stability: Explain why constraining policy changes matters in both robotic control and language-model fine-tuning.
Why it matters: Controlling the size of a policy change helps preserve useful behavior across both physical and language tasks.
Key terms: PPO, continuous control, state space, action space, policy gradient, trust-region constraint, KL, policy
11.8.2 Recommendation systems at scale
This section frames recommendations as a policy problem and connects exploration in large-scale recommendation systems to sampling diversity in GRPO.
- Recommendation Policy Components: Describe the state, action, and reward in a recommendation policy over user sessions.
Why it matters: This mapping shows how a recommendation session can be treated as a policy-based decision problem. - Exploration and Sampling Diversity: Explain why recommendation policies need exploration and how the section connects this need to sampling diversity in GRPO.
Why it matters: Exploration helps a policy find opportunities it would miss by repeatedly choosing the familiar option.
Key terms: policy, state, action, reward, engagement, exploration versus exploitation, sampling, diversity, gradient, temperature
11.8.3 Drug discovery and molecular design
This section applies sequence-level reinforcement learning and verifiable rewards to molecular design.
- Molecular sequence generation: Explain why molecular generation fits sequence-level reinforcement learning and what properties it can target.
Why it matters: Representing molecules as sequences lets policy-based methods guide their generation toward useful drug properties. - Docking verifier and bottleneck: Describe how molecular docking provides verifiable feedback and why the verifier is the bottleneck in this application.
Why it matters: The quality and availability of verification constrain how molecular-design RL can be applied.
Key terms: sequence-level RL, chemical grammar, binding affinity, toxicity, pharmacokinetics, verifier, molecular docking, binding affinity score, verifiable rewards, wet-lab assay
11.8.4 Code generation with execution feedback
This section explains how code execution supplies verifiable feedback for policy-gradient training of code generation models.
- Execution Feedback Training Pipeline: Trace how test-suite results are used to fine-tune a code generation model and describe how this adapts the FinQA pipeline.
Why it matters: Execution turns code correctness into feedback that can train a generation model. - Partial Code Feedback: Explain how compilation provides a useful feedback distinction even when generated code fails its tests.
Why it matters: Feedback can distinguish an unsuccessful but well-formed program from one that cannot be parsed.
Key terms: test-suite execution, reward signal, group-relative advantages, verifier function, format reward, compiles, fails to parse
11.8.5 Operations research and combinatorial optimization
This section describes how policy-based reinforcement learning can learn heuristics for large-scale combinatorial optimization and why those heuristics can benefit businesses.
- Sequential construction with learned policies: Explain how learned policies approach combinatorial optimization, using vehicle routing as an example.
Why it matters: This shows how policy learning can turn a large optimization problem into a sequence of decisions. - Business benefits of learned heuristics: Describe the potential business benefits of learned routing policies compared with classical solvers.
Why it matters: The value of a routing policy depends on both operational responsiveness and the savings its decisions produce.
Key terms: combinatorial problems, learned heuristics, attention mechanisms, sequential construction, PPO, REINFORCE, route length, learned policy, traffic conditions, demand shocks
11.8.6 Clinical decision support and treatment sequencing
This section illustrates policy-based reinforcement learning in clinical treatment and explains how shared optimization tools transfer across domains through domain-specific rewards and verification.
- Sequential Clinical Treatment: Describe how clinical treatment can be framed as sequential decision-making and what retrospective patient records have enabled in narrow protocols.
Why it matters: This shows how a clinical treatment pathway can become a sequential decision problem. - Clinical Safety Constraint: Explain why a clinical RL policy should remain close to the standard of care and how this resembles the KL penalty in LLM fine-tuning.
Why it matters: In clinical use, trustworthy limits on dangerous departures can matter more than average gains. - Transferable Optimization and Verification: Explain how policy-based optimization tools can transfer between domains and why a trustworthy domain-specific verifier matters.
Why it matters: Reusing training machinery is useful only when the new domain's feedback measures the right thing reliably.
Key terms: sequential, retrospective patient records, policies, safety constraint, KL penalty, standard of care, policy-based methods, baseline, verifier, reward design
11.9 Summary
Concepts explained in this chapter
- Group Relative Policy OptimizationA reinforcement-learning algorithm that compares rewards among multiple outputs sampled for the same prompt to estimate relative advantages for policy updates.
- reinforcement learning with verifiable rewardsA reinforcement-learning approach that uses objectively checkable criteria to reward model outputs.
- verifierA verifier is a procedure that checks a model's answer against a specified correctness criterion or reference answer.
- low-rank adaptationLoRA is a parameter-efficient fine-tuning technique that trains low-rank updates to a model's weights while keeping its original weights fixed.
- Schulman k3 estimatorA nonnegative Monte Carlo estimator of reverse Kullback–Leibler divergence computed from differences between log probabilities.
- group-relative baselineA baseline that compares a response’s reward with the rewards of other responses generated for the same prompt.
- gradient checkpointingGradient checkpointing reduces neural-network activation memory by recomputing activations during the backward pass instead of retaining them.
- reference modelA reference model is a fixed model whose policy distribution provides a comparison point during policy training.
- low-rank subspaceA low-rank subspace is a restricted set of directions whose dimensionality is smaller than that of the full space being represented.
- KL regularizationA regularization term based on Kullback–Leibler divergence that discourages a policy from moving too far from a reference policy.
- continuous controlContinuous control is decision-making in which an agent selects real-valued control signals rather than discrete actions.
- learning rateA learning rate is a numerical step-size parameter that controls how strongly an update changes an estimate or model parameter.
- reward functionA reward function is a mapping that assigns numerical feedback to states, actions, transitions, or outcomes in a decision process.
- convergenceConvergence is the approach of successive numerical estimates or policies toward a stable limiting result.
- policy gradient methodA reinforcement-learning method that directly optimizes the parameters of a policy using the gradient of expected return.
- format bonusAn additional reward for presenting an answer in a required format, independent of whether its content is correct.
- Monte Carlo simulationMonte Carlo estimation approximates a quantity by averaging values obtained from random samples.
- sparse rewardA sparse reward is a feedback signal provided at only a small number of points, rather than continuously throughout a task.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
