Reinforcement learning (RL) learns a policy: a rule that maps an observed state to a distribution over actions. The agent acts in an environment, receives rewards, and updates its policy to maximize expected cumulative reward. Unlike supervised learning from a fixed target output, RL improves behavior from the consequences of actions, often through trial and error. section 3.1 section 10.4
A trajectory records an episode as states, actions, and rewards. Its probability combines the initial-state distribution, policy action probabilities, and environment transition dynamics. The standard finite-horizon objective maximizes expected discounted return, Σₜ γᵗr(sₜ,aₜ), where γ balances immediate against future reward; continuing tasks use γ<1 to keep return well-defined. section 3.1
Equation 3.2
\[\label{eq:rl_opt} {\max_\pi \; \mathbb{E}_{\tau \sim p_{\pi}} \left[ \sum_{t=0}^{T-1} \gamma^t r(s_t, a_t) \right].}\]The objective makes the operational goal explicit: sample trajectories from the current policy, sum discounted rewards, and adjust the policy toward higher expected return. section 3.1
Thermostat control
A thermostat observes the room temperature, chooses whether to turn a heater on or off, and receives +1 when temperature is within 2°F of 70°F. Starting from near-random switching, it can learn through repeated episodes that heating when cold and stopping when warm earns more reward. The room’s heating and cooling physics are environment dynamics: actions influence them but do not define them. section 3.1.1
For language-model post-training, the policy generates a completion for a prompt and a reward model scores that completion. An RL optimizer then takes gradient steps on on-policy samples—outputs from the current model. A frozen reference model can supply a KL penalty, discouraging the tuned policy from drifting excessively while reward drives improvement. section 6.0 section 1.3
RL is used where success can be evaluated after acting, including:
- Human-preference optimization: instruction tuning establishes answer format, a reward model learns preferences over completions, and RL optimizes against its scalar scores. section 1.3 section 1.7
- Reasoning with verifiable rewards: recent RL-focused language-model training builds on pretrained models when task outcomes can be checked. section 7.2.1
- Multistep tool use: environment feedback such as task success or constraint satisfaction can train tool-augmented actions beyond formatting and tool selection. section 13.5
RL offers flexible, scalable updates and can target a narrow prompt distribution without necessarily erasing broad capabilities. It also demands substantially more infrastructure than instruction tuning or direct preference optimization, and training can be brittle, with instability, loss spikes, and crashes. Reward design matters: standard RL assumes a numerical objective, whereas preference models aggregate context-dependent human judgments. section 3.1.6 section 7.2.1 section 10.3
In post-training comparisons, RL samples from its own policy and can be expressed through reverse KL toward a reward-weighted reference policy, while supervised fine-tuning corresponds approximately to forward KL from a target distribution. Controlled studies cited here find RL improving out-of-distribution performance with more compute and retaining prior knowledge better across sequential tasks; these are empirical findings rather than universal guarantees. section 15.2.2 section 15.2.1
