Reinforcement learning (RL) is a way for an agent to learn decisions through trial and error. The agent acts in an environment, receives rewards or penalties, and refines a policy—a way of choosing actions—to maximize cumulative reward. This makes RL useful when an optimization problem can be approached as a sequence of decisions. section 12.1 section 12.7
First decide whether actions affect what happens next:
- In a Markov decision process (MDP), actions change the environment and its future states. Decisions must account for consequences across time steps. section 12.1
- In a multi-armed bandit (MAB), the agent repeatedly chooses among arms to earn reward, without modeling how a choice changes future options. A contextual bandit also uses current side information; the best arm may change with that context. section 12.1 section 12.7
Choosing the right formulation
Suppose an agent repeatedly selects one of several options. If each selection yields reward without changing the options available next time, model the choice as a bandit. If a selection changes the next state the agent must navigate, use an MDP formulation. The distinction determines whether immediate choices alone capture the problem. section 12.1 section 12.7
RL methods can also be classified by whether they use a model of environment transitions and rewards, and by whether policy updates are on-policy or off-policy. Advantage actor-critic (A2C) and proximal policy optimization (PPO) are model-free, on-policy methods. PPO clips its update objective to limit large policy changes and support more stable learning; that is a design rationale, not a promise of an optimal result. section 12.7
