Markov decision process
A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action.
Applied Reinforcement Learning
A Markov decision process is a formal model of sequential decision-making in which an agent selects actions in states, receives rewards, and causes probabilistic transitions whose relevant future behavior depends on the current state.
- How it is used
- Frames business optimization as sequential decisions under uncertainty.
- How deep it goes
- Develops states, transitions, rewards, policies, and simulations, then implements policy iteration for resource allocation.
- Examples
- Uses dynamic pricing, factory production planning, and project resource allocation to map business variables into MDPs.
- What you can do after
- Build it from scratch by defining an MDP and simulating policies for a production-planning problem.
Read how Applied Reinforcement Learning explains Markov decision process →
Optimization Algorithms: AI techniques for design, planning, and control problems
A Markov decision process is a model of sequential decision-making defined by states, actions, transition probabilities, and rewards with future dynamics dependent only on the current state and action.
- How it is used
- Frames sequential decisions under uncertainty as reinforcement learning and combinatorial optimization problems.
- How deep it goes
- Defines states, actions, transitions, rewards, and policies, and explains their role in agent-environment interaction.
- Examples
- Illustrates uncertain transitions with driving and biking commute choices, and cites vehicle routing applications.
- What you can do after
- Evaluate sequential decision problems by identifying states, actions, transitions, rewards, and a policy.
Reinforcement Learning from Human Feedback
A Markov decision process is a mathematical framework that models states, actions, rewards, and probabilistic transitions for sequential decision making under the Markov property.
- How it is used
- Frames reinforcement learning as sequential decision making over states, actions, rewards, and transitions.
- How deep it goes
- Defines finite-MDP objectives, returns, value functions, and rollout estimates, then connects them to policy-gradient updates.
- Examples
- Uses RLHF prompts and completions, token-level actions and prefixes, PPO advantages, and sampled trajectories as concrete formulations.
- What you can do after
- Evaluate/understand how RL objectives represent trajectories, estimate returns, weight states, and update policies from sampled experience.
Read how Reinforcement Learning from Human Feedback explains Markov decision process →
Transformers in Action
A Markov decision process is a mathematical model for sequential decision-making in which an agent selects actions in states, receives rewards, and undergoes probabilistic transitions.
- How it is used
- Frames reinforcement-learning alignment as sequential decision-making before introducing proximal policy optimization.
- How deep it goes
- Defines stochastic control, five components, state-based policies, uncertainty, and reward maximization without developing PPO.
- Examples
- Uses portfolio construction and a dog-training obstacle course to illustrate uncertain sequential decisions and reward maximization.
- What you can do after
- Evaluate sequential decision problems by identifying agents, states, actions, rewards, policies, and uncertain transitions.
Read how Transformers in Action explains Markov decision process →
Causal AI
A Markov decision process is a sequential decision model in which an agent selects actions in states and receives rewards, with future dynamics determined by the current state and action.
- How it is used
- Models sequential decisions in reinforcement learning and supports causal reasoning about state transitions under chosen actions.
- How deep it goes
- Defines states, actions, transitions, rewards, and the Markov assumption, then identifies the transition function as a causal kernel.
- Examples
- Uses a 3 × 4 grid world with movement actions, an obstacle, positive and negative terminal rewards, and random starting positions.
- What you can do after
- Evaluate sequential decision problems by specifying states, actions, transitions, rewards, and their causal DAG representation.
Timeless Algorithms, The Foundational Ideas
A Markov decision process is a model of sequential decisions in which actions influence transitions among states and rewards, with future dynamics depending on the current state and action.
- How it is used
- Models sequential optimization when actions can lead to uncertain outcomes.
- How deep it goes
- Defines probabilistic transitions, rewards, and recursive value relationships without deriving a numerical example.
- Examples
- Illustrates uncertain transitions, rewards, and future-state value abstractly rather than through a worked scenario.
- What you can do after
- Evaluate actions by balancing immediate rewards against expected value in future states.
Read how Timeless Algorithms, The Foundational Ideas explains Markov decision process →
Also covered in
What to know first
- reward signalA reward signal is a value or feedback indicating the desirability, benefit, or cost associated with an action, outcome, or behavior.
- stateState is information describing the condition or configuration of a system or computation at a particular time that may influence its behavior or subsequent transitions.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- actionAn action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
Related concepts
- reinforcement learningused by · Reinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- Markov chaincontrasts with · A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history.
- Bellman equationuses · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
- actionhas part · An action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
- statehas part · State is information describing the condition or configuration of a system or computation at a particular time that may influence its behavior or subsequent transitions.
Questions about Markov decision process
What is Markov decision process?
A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action.
Which Manning books explain Markov decision process?
Applied Reinforcement Learning, Optimization Algorithms: AI techniques for design, planning, and control problems, Reinforcement Learning from Human Feedback, Transformers in Action, Causal AI and Timeless Algorithms, The Foundational Ideas.
What should I know before learning Markov decision process?
Manning books that teach Markov decision process usually introduce reward signal, state, reinforcement learning and action first.
What is the difference between Markov decision process and Markov chain?
A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action. A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history.
What is Markov decision process a kind of?
Markov decision process is a kind of stochastic control process. A stochastic control process is a mathematical model of a system evolving over time under both controlled actions and random events.
What are the parts of Markov decision process?
Markov decision process includes action and state.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





