liveBook · Manning

Markov chain

A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history.

How 7 Manning books explain it

Statistics Every Programmer Needs

Markov analysis is a technique for modeling transitions among a finite set of states using transition probabilities to predict future states.

How it is used
Forecasts customer switching, market shares, and loan-delinquency movements from current states and transition probabilities.
How deep it goes
Works through state definitions, transition matrices, NumPy forecasts, equilibrium, and absorbing states without pursuing abstract proofs.
Examples
Uses four-carrier market shares, a running-or-not-running engine, and loan-delinquency categories.
What you can do after
Apply it via NumPy to calculate one- and two-period state distributions and interpret longer-run behavior.

Read how Statistics Every Programmer Needs explains Markov chain →

Fabulous Adventures in Data Structures and Algorithms

A Markov process is a stochastic process in which the conditional distribution of the next state depends only on the current state.

How it is used
Models random state sequences for text generation and serves as the sequence structure used by the Metropolis algorithm.
How deep it goes
Defines the current-state rule and builds a generic sampler and count-based word model, then reuses the sampler for Metropolis transitions.
Examples
Uses a gambler’s chip-count walk and a word-pair generator trained on Shakespearean sentences.
What you can do after
Build it from scratch as a C# distribution of state sequences, with transitions that can also terminate a sequence.

Read how Fabulous Adventures in Data Structures and Algorithms explains Markov chain →

Applied Reinforcement Learning

A Markov chain is a stochastic model describing how a system evolves among states when the next state depends probabilistically on the current state.

How it is used
Models known stochastic transitions as a foundation for reasoning about systems with unknown transitions and reinforcement-learning decisions.
How deep it goes
Explains the Markov-property shortcut and demonstrates simulation and empirical transition estimates, without treating the chain as an optimization model.
Examples
Uses software-service customer states, a transition graph and matrix, and simulated next-state frequencies.
What you can do after
Apply it via NumPy to simulate state transitions and estimate next-state distributions from observed frequencies.

Read how Applied Reinforcement Learning explains Markov chain →

Machine Learning Algorithms in Depth

A Markov model is a stochastic model in which the conditional distribution of future states depends on the present state rather than the full history.

How it is used
Provides a transition-based model for page rank, posterior sampling, and latent-state time series.
How deep it goes
Defines first-order dependence and stochastic matrices, then develops state-distribution updates and long-term behavior; detailed sampling algorithms are deferred.
Examples
Uses a two-state transition diagram and a web-page link graph to illustrate transition probabilities.
What you can do after
Evaluate/understand it: form a discrete transition matrix, update state distributions, and interpret a stationary distribution in page rank or MCMC.

Read how Machine Learning Algorithms in Depth explains Markov chain →

Grokking Bayes

A Markov chain is a stochastic process in which the distribution of the next state depends on the current state rather than the full preceding history.

How it is used
Supports MCMC by providing a dependent walk that efficiently explores high-density regions of an unnormalized posterior.
How deep it goes
Defines state dependence and explains proposal and accept/reject mechanisms, but defers the mathematical proof of convergence.
Examples
Uses Beta(4,3) samples, ordered MCMC walks, grid approximations, and a tourist-attractions analogy.
What you can do after
Evaluate how dependent MCMC samples can still approximate a target posterior distribution.

Read how Grokking Bayes explains Markov chain →

Optimization Algorithms: AI techniques for design, planning, and control problems

A stochastic model representing a sequence of states in which each transition probability depends only on the current state.

How it is used
Models memoryless state transitions for stochastic optimization and other probabilistic processes.
How deep it goes
Defines the memoryless property and illustrates transitions, but does not develop formal probability theory.
Examples
Illustrates transitions among states A, B, and C and probabilistic acceptance of neighboring solutions during simulated annealing.
What you can do after
Apply it via SymPy's DiscreteMarkovChain for finite discrete-time homogeneous models.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains Markov chain →

Timeless Algorithms, The Foundational Ideas

A Markov chain is a stochastic model of transitions among states in which the next state depends only on the current state.

How it is used
Demonstrates how the multiplication rule combines conditional probabilities for dependent transitions over time.
How deep it goes
Defines the current-state dependence and explains sequence probabilities through successive conditional probabilities.
Examples
Uses customer behavior, brand migration, website navigation, weather forecasting, and market share analysis as applications.
What you can do after
Understand how conditional transition probabilities combine to describe a system's evolution.

Read how Timeless Algorithms, The Foundational Ideas explains Markov chain →

Also covered in

What to know first

Related concepts

Questions about Markov chain

What is Markov chain?

A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history.

Which Manning books explain Markov chain?

Statistics Every Programmer Needs, Fabulous Adventures in Data Structures and Algorithms, Applied Reinforcement Learning, Machine Learning Algorithms in Depth, Grokking Bayes, Optimization Algorithms: AI techniques for design, planning, and control problems and Timeless Algorithms, The Foundational Ideas.

What should I know before learning Markov chain?

Manning books that teach Markov chain usually introduce conditional probability, data exploration, linear regression and normal distribution first.

What is the difference between Markov chain and Markov decision process?

A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history. A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action.

What kinds of Markov chain are there?

Kinds of Markov chain include time-invariant Markov chain.

What is Markov chain a kind of?

Markov chain is a kind of stochastic process. A stochastic process is a collection of random variables indexed by time, space, or another parameter that represents the evolution of a random phenomenon.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.