11 Teach LLMs to reason with Verifiable Feedback using GRPO
This chapter covers
- Why deterministic verifiers replace reward models for tasks with ground-truth answers
- LoRA: a new way to fine-tune billion-parameter models
- GRPO: removing the value network by using a group of sampled answers as the baseline
- Building a complete GRPO, LoRA, and RLVR pipeline from scratch for the financial domain
The art of knowing is knowing what to ignore.
Rumi, 13th century Persian poet and Sufi
Not every task we ask a language model to perform is a matter of taste. Ask it to draft a marketing email, and there is no single right answer — different readers want different tones. Ask it to compute a company's year-over-year change in net revenue from a financial report, and there is exactly one correct number. The model either gets it, or it doesn't.
That second kind of task — one with a ground-truth answer that a program can check without a human in the room — is what this chapter is about. When correctness can be verified mechanically, the entire reward-modeling stage of chapter 10's RLHF pipeline becomes unnecessary: we don't need to collect human preference pairs or train a second neural network to guess at what people want. We just need a function, which we call the verifier, that checks the model's answer against the ground truth.
Three ideas make this approach work at scale, and this chapter takes them up one at a time, in this order: