Reinforcement learning (RL) trains an agent to choose policies that maximize cumulative reward. Instead of receiving explicit action labels, the agent learns from consequences, adjusting behavior through rewards and losses. section 12.5 section 12.0
A policy maps information about a decision problem to an action. RL searches for policies that optimize outcomes over time, making it a sequential decision-making problem with probabilistic consequences: even an optimal action can produce an unfavorable result by chance. section 8.1.5 section 12.0
A causal framing changes what the learner treats as evidence:
- Distinguish actions that correlate with high rewards from actions that cause higher rewards. section 1.4.2
- Assign credit to actions by asking how much reward would have resulted under a different policy. section 1.4.2
- Compare actual outcomes with simulated outcomes under alternative policies to quantify regret and learn from conditions not normally observed. section 8.1.5
The conventional and causal Bellman formulations correspond to conditioning on an observed action versus intervening on that action. Confounder, mediator, and collider biases can affect conventional formulations; causal approaches matter when they change which policy appears optimal. Policies behave like stochastic interventions, and MDPs or POMDPs can be represented as causal DAGs for graphical causal inference. section 12.7 section 12.7
Counterfactual policy evaluation
An investment agent can backtest alternative portfolio-allocation policies as new prices arrive. A game-playing agent can replay saved games under a different policy, compare the simulated and actual outcomes, and use the difference to improve its policy. section 8.1.5 section 8.1.5
