chapter eleven

11 Teach LLMs to reason with Verifiable Feedback using GRPO

 

This chapter covers

  • Why deterministic verifiers replace reward models for tasks with ground-truth answers
  • LoRA: a new way to fine-tune billion-parameter models
  • GRPO: removing the value network by using a group of sampled answers as the baseline
  • Building a complete GRPO, LoRA, and RLVR pipeline from scratch for the financial domain
The art of knowing is knowing what to ignore.

Rumi, 13th century Persian poet and Sufi

Not every task we ask a language model to perform is a matter of taste. Ask it to draft a marketing email, and there is no single right answer — different readers want different tones. Ask it to compute a company's year-over-year change in net revenue from a financial report, and there is exactly one correct number. The model either gets it, or it doesn't.

That second kind of task — one with a ground-truth answer that a program can check without a human in the room — is what this chapter is about. When correctness can be verified mechanically, the entire reward-modeling stage of chapter 10's RLHF pipeline becomes unnecessary: we don't need to collect human preference pairs or train a second neural network to guess at what people want. We just need a function, which we call the verifier, that checks the model's answer against the ground truth.

Three ideas make this approach work at scale, and this chapter takes them up one at a time, in this order:

11.1 Teaching language models to reason on financial reports

11.2 Reinforcement Learning with Verifiable Rewards

11.3 LoRA: fine-tuning large models with tiny adapters

11.4 GRPO: group relative policy optimization

11.5 Training stability in GRPO

11.5.1 Memory for logit computation

11.5.2 Computing the reference policy without a second copy

11.5.3 Gradient checkpointing

11.5.4 Diagnosing training stability with KL divergence

11.6 Five lessons from training GRPO at scale

11.6.1 Lesson 1: Learning rate

11.6.2 Lesson 2: Too hot RL vs too cold RL

11.6.3 Lesson 3: KL penalty

11.6.4 Lesson 4: Base model impact

11.6.5 Lesson 5: KL signal is the key

11.7 Building the financial reasoning agent

11.7.1 Pipeline configuration

11.7.2 LoRA injection

11.7.3 FinQA data pipeline and prompt formatting

11.7.4 RLVR verifier: extracting and scoring answers

11.7.5 GRPO training: rollouts, advantages, and the update step

11.7.6 Evaluation: comparing base and fine-tuned models

11.8 Policy-based methods beyond language models

11.8.1 Robotics and physical control

11.8.2 Recommendation systems at scale