Markov decision process

As explained in Causal AI

A Markov decision process is a sequential decision model in which an agent selects actions in states and receives rewards, with future dynamics determined by the current state and action.

A Markov decision process (MDP) models sequential decisions as an agent acting in an environment. Each state describes the current situation, each action can change that state, and rewards express desirable or costly outcomes. section 12.5.1

Grid-world navigation

In a 3 × 4 grid, the agent can move up, down, left, or right. Reaching {0, 3} ends the game with reward 100; entering {1, 3} produces −100, while {1, 1} is an impassable obstacle. The agent must choose a route from a random starting cell. section 12.5.1

The transition function gives the probability of the next state conditional on the current state and chosen action. The Markov assumption says that the current state does not depend on the full prior history once the previous state is known. section 12.5.1 section 12.5.2

Figure 12.27 A simple grid world

Use the grid-world figure to trace legal actions, obstacles, terminal rewards, and routes through states. section 12.5.1

Figure 12.28 The Markov decision process represented as a DAG

Use the causal DAG to connect states, actions, rewards, terminal states, and the transition function as a causal Markov kernel. section 12.5.2

How Causal AI covers it

How deep it goes

Defines states, actions, transitions, rewards, and the Markov assumption, then identifies the transition function as a causal kernel.

Examples

Uses a 3 × 4 grid world with movement actions, an obstacle, positive and negative terminal rewards, and random starting positions.

What it assumes

Assumes an agent, environment, and observable state, while introducing the MDP components through a concrete navigation problem.

What you can do after

Evaluate sequential decision problems by specifying states, actions, transitions, rewards, and their causal DAG representation.

How it connects

Connects reinforcement learning, causal DAGs, d-separation, do-calculus, terminal states, and transition functions.

liveBookmanning.com

Read Causal AI on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA Markov decision process is a model of sequential decision-making defined by states, actions, transition probabilities, and rewards with future dynamics dependent only on the current state and action.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA Markov decision process is a mathematical framework that models states, actions, rewards, and probabilistic transitions for sequential decision making under the Markov property.
• 

  liveBookmanning.com
  Transformers in ActionA Markov decision process is a mathematical model for sequential decision-making in which an agent selects actions in states, receives rewards, and undergoes probabilistic transitions.

Compare how 4 books explain Markov decision process →

Related concepts

• reinforcement learningused by · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• Markov chaincontrasts with · A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history.
• Bellman equationuses · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
• statehas part · State is information or a configuration describing the condition of a system, entity, or computation at a particular time and potentially influencing subsequent behavior or decisions.
• reward signaluses · A reward signal is an evaluative feedback value representing the desirability or payoff of an action, outcome, or state.

This explainer was generated by Manning's knowledge graph from the text of Causal AI. The section links point to the parts of the book it draws on.