liveBook · Manning

Markov decision process

Also called MDPs

A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action.

How 6 Manning books explain it

Applied Reinforcement Learning

A Markov decision process is a formal model of sequential decision-making in which an agent selects actions in states, receives rewards, and causes probabilistic transitions whose relevant future behavior depends on the current state.

How it is used
Frames business optimization as sequential decisions under uncertainty.
How deep it goes
Develops states, transitions, rewards, policies, and simulations, then implements policy iteration for resource allocation.
Examples
Uses dynamic pricing, factory production planning, and project resource allocation to map business variables into MDPs.
What you can do after
Build it from scratch by defining an MDP and simulating policies for a production-planning problem.

Read how Applied Reinforcement Learning explains Markov decision process →

Optimization Algorithms: AI techniques for design, planning, and control problems

A Markov decision process is a model of sequential decision-making defined by states, actions, transition probabilities, and rewards with future dynamics dependent only on the current state and action.

How it is used
Frames sequential decisions under uncertainty as reinforcement learning and combinatorial optimization problems.
How deep it goes
Defines states, actions, transitions, rewards, and policies, and explains their role in agent-environment interaction.
Examples
Illustrates uncertain transitions with driving and biking commute choices, and cites vehicle routing applications.
What you can do after
Evaluate sequential decision problems by identifying states, actions, transitions, rewards, and a policy.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains Markov decision process →

Reinforcement Learning from Human Feedback

A Markov decision process is a mathematical framework that models states, actions, rewards, and probabilistic transitions for sequential decision making under the Markov property.

How it is used
Frames reinforcement learning as sequential decision making over states, actions, rewards, and transitions.
How deep it goes
Defines finite-MDP objectives, returns, value functions, and rollout estimates, then connects them to policy-gradient updates.
Examples
Uses RLHF prompts and completions, token-level actions and prefixes, PPO advantages, and sampled trajectories as concrete formulations.
What you can do after
Evaluate/understand how RL objectives represent trajectories, estimate returns, weight states, and update policies from sampled experience.

Read how Reinforcement Learning from Human Feedback explains Markov decision process →

Transformers in Action

A Markov decision process is a mathematical model for sequential decision-making in which an agent selects actions in states, receives rewards, and undergoes probabilistic transitions.

How it is used
Frames reinforcement-learning alignment as sequential decision-making before introducing proximal policy optimization.
How deep it goes
Defines stochastic control, five components, state-based policies, uncertainty, and reward maximization without developing PPO.
Examples
Uses portfolio construction and a dog-training obstacle course to illustrate uncertain sequential decisions and reward maximization.
What you can do after
Evaluate sequential decision problems by identifying agents, states, actions, rewards, policies, and uncertain transitions.

Read how Transformers in Action explains Markov decision process →

Causal AI

A Markov decision process is a sequential decision model in which an agent selects actions in states and receives rewards, with future dynamics determined by the current state and action.

How it is used
Models sequential decisions in reinforcement learning and supports causal reasoning about state transitions under chosen actions.
How deep it goes
Defines states, actions, transitions, rewards, and the Markov assumption, then identifies the transition function as a causal kernel.
Examples
Uses a 3 × 4 grid world with movement actions, an obstacle, positive and negative terminal rewards, and random starting positions.
What you can do after
Evaluate sequential decision problems by specifying states, actions, transitions, rewards, and their causal DAG representation.

Read how Causal AI explains Markov decision process →

Timeless Algorithms, The Foundational Ideas

A Markov decision process is a model of sequential decisions in which actions influence transitions among states and rewards, with future dynamics depending on the current state and action.

How it is used
Models sequential optimization when actions can lead to uncertain outcomes.
How deep it goes
Defines probabilistic transitions, rewards, and recursive value relationships without deriving a numerical example.
Examples
Illustrates uncertain transitions, rewards, and future-state value abstractly rather than through a worked scenario.
What you can do after
Evaluate actions by balancing immediate rewards against expected value in future states.

Read how Timeless Algorithms, The Foundational Ideas explains Markov decision process →

Also covered in

What to know first

Related concepts

Questions about Markov decision process

What is Markov decision process?

A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action.

Which Manning books explain Markov decision process?

Applied Reinforcement Learning, Optimization Algorithms: AI techniques for design, planning, and control problems, Reinforcement Learning from Human Feedback, Transformers in Action, Causal AI and Timeless Algorithms, The Foundational Ideas.

What should I know before learning Markov decision process?

Manning books that teach Markov decision process usually introduce reward signal, state, reinforcement learning and action first.

What is the difference between Markov decision process and Markov chain?

A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action. A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history.

What is Markov decision process a kind of?

Markov decision process is a kind of stochastic control process. A stochastic control process is a mathematical model of a system evolving over time under both controlled actions and random events.

What are the parts of Markov decision process?

Markov decision process includes action and state.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.