liveBook · Manning

Reinforcement learning

As explained in LLM Customization and Fine-Tuning

A machine-learning approach in which an agent learns a policy through reward signals from interactions or sampled outcomes.

In preference optimization, reinforcement learning (RL) rewards outputs that lead to correct answers. Behaviors absent from Stage 1 supervised fine-tuning (SFT) can emerge when verification improves correctness. section 8.6.2

RL can support enterprise reasoning tasks, but its use should match the task:

  • For multi-step debugging, compliance checks, and root-cause analysis, reward-driven verification can help models trace beyond surface symptoms. section 8.6.2
  • For routine classification, extraction, or generation, prefer DPO on a well-curated SFT model; full emergent reasoning is usually unnecessary. section 8.6.2
  • To borrow the reasoning-training pattern, use a small Stage-1 SFT for task formats, then try small-scale GRPO on a narrow, high-value domain slice. section 8.6.2
Incident: users report login failures starting around 02:14.
 
The obvious read is "authentication failed," but let me check what else changed near 02:14.
  - Auth service logs: errors begin at 02:14 (the symptom).
  - Database logs: write timeouts start at 02:13, one minute earlier.
  - Host metrics: disk usage reaches 100% at 02:12, just before the timeouts.
Wait, let me order these: 02:12 disk full -> 02:13 DB write timeouts -> 02:14 logins fail.
Could the auth service itself be the cause? If so, its errors should predate the disk filling, but the earliest anomaly is the disk at 02:12, so auth is downstream, not root.
Verify the chain: a full disk stops the database from committing, so session tokens cannot persist, so logins fail. The evidence lines up end to end.
Root cause: the full disk, not the authentication service.
Fix: reclaim or extend the volume, then confirm token writes and logins recover.

Trace how a reasoning-trained model checks an incident timeline, tests an alternative cause, and verifies the root cause. section 8.6.2

RL and small adapters

A reported TinyLoRA result fine-tunes Qwen2.5-7B with 13 parameters using GRPO, reaching 91% GSM8K accuracy versus a 76% baseline; SFT barely improves at that parameter budget. section 5.5

How LLM Customization and Fine-Tuning covers it

How deep it goes
Explains behavior emergence through GRPO and recommends narrow applications, while treating DeepSeek-style training as a case study rather than a default.
Examples
Uses login-failure root-cause analysis and GSM8K math reasoning to illustrate verification behavior and parameter-efficient training.
What you can do after
Evaluate whether to reserve GRPO for narrow, high-value reasoning tasks instead of routine classification, extraction, or generation.
How it connects
Connects reinforcement learning with GRPO, supervised fine-tuning, direct preference optimization, and reasoning-task adaptation.

Also explained in

Compare how 28 books explain reinforcement learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.