An agent choosing an action needs more than a destination: it needs a model of how the environment changes. A transition function gives the probability of reaching each next state after taking an action in the current state. section 12.5.1
For discrete states, the transition probability is written P(Sₜ₊₁ = s′ | Sₜ = s, Aₜ = a). It answers: given current state s and action a, how likely is next state s′? Actions can lead to different outcomes, including remaining in place when a border or obstacle blocks movement. section 12.5.1
Grid-world navigation
In a grid world, the agent moves up, down, right, or left toward a reward of 100 while avoiding a reward of −100 and an obstacle. From a particular cell, the transition probability describes where the agent will be after each selected move, supporting a policy that reaches the goal without entering the loss state. section 12.5.1
The probability also has a causal interpretation. In the MDP’s causal DAG, the direct parents of Sₜ₊₁ are Sₜ and Aₜ, so the transition function is the causal Markov kernel for the next state. This makes the Markov assumption operational: the current state summarizes the relevant history for predicting the next state. section 12.5.2
For counterfactual reasoning, the transition probability must reflect the environment’s actual state-change rules. An SCM can encode those rules through assignment functions, but specifying correct counterfactual distributions is difficult when the underlying mechanisms are unknown. Rule-based games make this easier because their rules support simulation of alternative action trajectories. section 12.6.3
Figure 12.27 A simple grid world
Use the grid-world figure to connect states, actions, obstacles, terminal rewards, and possible transitions. section 12.5.1
Figure 12.28 The Markov decision process represented as a DAG
Use the DAG to trace how the previous state and action determine the next state’s causal Markov kernel. section 12.5.2
