A state represents an agent’s current situation or context within an environment. It captures the information needed to describe where the agent is in a sequential decision process and can be discrete, such as a grid cell, or continuous. section 12.5.1
Grid-world position
In a 3-by-4 grid world, each reachable cell is a state. The agent starts in a random nonterminal cell, navigates around an obstacle, avoids a loss square, and seeks a goal square. Actions change the current state, although borders and obstacles can leave it unchanged. section 12.5.1
Within a Markov decision process, the current state summarizes the relevant decision context: given the last state, the full earlier history is independent of the current state. The next state depends causally on the previous state and selected action through the transition function P(Sₜ₊₁ = s′ | Sₜ = s, Aₜ = a). section 12.5.2 section 12.5.2
State representations can differ according to what the agent can observe:
- Fully observed MDP: the agent knows which state it occupies. section 12.5.1
- Partially observed MDP: the agent infers its state from incomplete environmental evidence. section 12.5.3
- Latent-state models can distinguish an unobserved state causing observations from an unobserved common cause affecting behavior and utility. section 12.5.3
Figure 12.27 A simple grid world
Use the grid-world figure to inspect how positions, obstacles, terminal cells, and rewards instantiate states in a concrete environment. section 12.5.1
Figure 12.28 The Markov decision process represented as a DAG
Use the DAG figure to trace how the previous state and action determine the next state in an MDP. section 12.5.2
