liveBook · Manning

Markov property

Also called Markov assumption

The Markov property is the condition that, given the current state, the distribution of future states is independent of the process’s earlier history.

How 4 Manning books explain it

Applied Reinforcement Learning

The Markov property is the condition that the distribution of future states and rewards depends on the current state and action rather than on unrepresented earlier history.

How it is used
Uses the property to guide state design for sequential business decisions under uncertainty.
How deep it goes
Explains the assumption through conditional transitions and state-design failures, then demonstrates simulation and a production-planning model.
Examples
Uses customer-state transitions, pricing, workforce scheduling, maintenance, and a scooter-and-bike production model.
What you can do after
Evaluate state designs for omitted information that could make future outcomes depend on unrepresented history.

Read how Applied Reinforcement Learning explains Markov property →

Causal AI

A Markov assumption is the assumption that each variable in an ordered chain depends only on the variable immediately preceding it, conditional on that predecessor.

How it is used
Simplifies sequential probability models and supports causal reasoning in graphical models and Markov decision processes.
How deep it goes
Explains the factorization simplification, its approximation risk, and its transition-kernel form, without deriving estimation procedures.
Examples
Uses the replacement P(z | x, y) with P(z | y) and an MDP transition P(Sₜ₊₁ = s′ | Sₜ = s, Aₜ = a).
What you can do after
Apply it via simplified conditional distributions and MDP transition functions while recognizing the assumption’s possible approximation error.

Read how Causal AI explains Markov property →

Build a Robo-Advisor with Python (From Scratch)

The Markov property is the condition that the probabilities of future states depend on the current state and action but not on earlier history.

How it is used
Frames financial planning as a reinforcement-learning problem whose transition probabilities use current wealth and asset allocation.
How deep it goes
Defines the history-independence assumption and applies it to wealth transitions, while deferring transition-probability implementation to subsequent code.
Examples
Uses chessboard positions and wealth under asset allocation to illustrate history-independent transitions.
What you can do after
Builds transition models that calculate next-period wealth probabilities from current wealth and selected asset allocation.

Read how Build a Robo-Advisor with Python (From Scratch) explains Markov property →

Optimization Algorithms: AI techniques for design, planning, and control problems

The property of a stochastic process whereby the distribution of the next state depends only on the current state and not on earlier states.

How it is used
Explains state transitions in simulated annealing and other stochastic models where only the current state determines the next move.
How deep it goes
Defines memorylessness, illustrates transition probabilities, and connects the idea to implementation without developing a formal proof.
Examples
Uses transitions among states A, B, and C plus improving and non-improving candidate solutions in simulated annealing.
What you can do after
Evaluate/understand whether a stochastic state transition depends only on the currently observed state.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains Markov property →

Also covered in

What to know first

Related concepts

Questions about Markov property

What is Markov property?

The Markov property is the condition that, given the current state, the distribution of future states is independent of the process’s earlier history.

Which Manning books explain Markov property?

Applied Reinforcement Learning, Causal AI, Build a Robo-Advisor with Python (From Scratch) and Optimization Algorithms: AI techniques for design, planning, and control problems.

What should I know before learning Markov property?

Manning books that teach Markov property usually introduce reward signal, state, Markov chain and sequential decision-making first.

What is Markov property part of?

Markov property is part of n-gram model. An n-gram model is a probabilistic language model that estimates sequence probabilities using contiguous sequences of a fixed number of tokens, typically to predict a token from preceding context.

What is Markov property also called?

Markov property is also called Markov assumption.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.