A causal reinforcement learning algorithm chooses actions by asking which actions produce rewards, not merely which actions correlate with high rewards. It uses causal relationships and counterfactual reasoning to identify effects and assign credit when outcomes depend on sequential decisions. section 1.4.2 section 12.0
A practical causal-RL workflow makes the decision process explicit:
- Represent an MDP or POMDP as a causal DAG, adapting a sequential-decision template to the problem rather than treating the template as universal. section 12.7
- Model policies as stochastic interventions. Their dependence on observed past states can still introduce backdoor confounding, so causal identification remains necessary. section 12.7
- Use counterfactual policy evaluation or counterfactual regret minimization to ask what reward a different policy would have produced. section 1.4.2 section 12.7
- When transition rules are known, construct a structurally causal model faithful to those rules for counterfactual decision-making. section 12.7
Policy comparison
An agent may observe that one action accompanies high rewards most often. Causal analysis instead compares the reward under an intervention on the policy, including the counterfactual reward under a different policy, while accounting for confounding from earlier observed states. section 1.4.2 section 12.7
