Reinforcement learning trains an agent through trial-and-error interaction with an environment. The agent generates actions, receives scalar rewards, and updates its policy to increase expected cumulative reward. section 7.3.1
The language-model training loop consists of four parts:
- Policy: a parameterized model that maps the current state or prompt to an action or response. section 7.3.1
- Reward: a scalar score for a prompt-response pair; higher scores encourage the policy to produce similar responses more often. section 7.3.1
- Trajectory: a rollout containing the prompt, generated steps, final response, and observed reward. section 7.3.1
- Training step: an update that nudges model parameters toward higher expected reward across a batch of trajectories. section 7.3.1
Unlike pretraining and supervised fine-tuning, which learn from next-token targets or demonstrations, reinforcement learning samples multiple candidates and favors those receiving higher rewards. This enables behavior optimization when no single correct demonstration exists. section 7.3
Classical environments can supply rewards directly, as in game scores or checkmate. Language models have no native score for a generated paragraph, so rewards must be engineered through fixed metrics, preference models, or programmatic verifiers. section 7.3.2
Financial assistant
A policy drafts a Q3 earnings summary. A reward function scores faithfulness, source citation, and utility, then an optimizer updates the model so similar prompts are more likely to produce high-scoring summaries. section 7.3.1
Figure 7.2 The reinforcement-learning (RL) loop. The agent learns to make decisions by acting in an environment, observing a state and reward, and updating its policy to maximize cumulative reward.
Visualizes the agent-environment loop: acting, observing state and reward, and updating the policy to maximize cumulative reward. section 7.3



























