A policy, π(a|s), maps an environment state to a probability distribution over actions. An agent samples actions from that distribution; together with the initial-state distribution and environment dynamics, those choices induce a distribution over complete trajectories. Reinforcement learning optimizes the policy for expected discounted reward. section 3.1
Equation 3.1
\[\label{eq:rl_dynam} {p_{\pi}(\tau)=\rho_0(s_0)\prod_{t=0}^{T-1}\pi(a_t\mid s_t)p(s_{t+1}\mid s_t,a_t).}\]The trajectory distribution separates what the policy controls—action probabilities—from what the environment supplies—initial states and transitions. section 3.1
In deep reinforcement learning, a neural network learns the policy. Policy-gradient methods optimize a parameterized policy πθ directly: the gradient of log πθ(a|s) identifies parameter changes that make an action more likely, while a scalar score determines whether that action should be reinforced or discouraged. For language models, states and actions can be written as prompts and completions; some implementations treat a whole completion as one action. section 6.1 section 6.2
A policy sits in the RL loop alongside:
- State: the current situation supplied to the agent. section 3.1
- Action: a sampled decision whose likelihood comes from the policy. section 3.1
- Reward and return: signals used to assess policy behavior over a trajectory, with discounting balancing immediate and future reward. section 3.1 section 6.2
- Dynamics: environment transitions influenced by actions but not controlled directly by the policy. section 3.1
Thermostat policy
For a room at 65°F, a policy can assign high probability to turning the heater on; above 70°F, it can assign high probability to turning it off. Trial and error updates an initially random policy toward behavior that keeps temperature near the target. CartPole follows the same pattern: corrective left or right pushes are learned to keep the pole balanced longer. section 3.1.1 section 3.1.2
In RLHF, the policy is the model being trained. PPO, GRPO, and REINFORCE are policy-gradient approaches that use recently generated samples to update it. Preference data is often collected on-policy: from the current model or a closely related checkpoint, so its generations better match the behavior being optimized. section 6.1 section 11.2
