Markov chain
A Markov chain is a stochastic process of states whose transition probabilities depend on the current state rather than the full preceding history.
Statistics Every Programmer Needs
Markov analysis is a technique for modeling transitions among a finite set of states using transition probabilities to predict future states.
- How it is used
- Forecasts customer switching, market shares, and loan-delinquency movements from current states and transition probabilities.
- How deep it goes
- Works through state definitions, transition matrices, NumPy forecasts, equilibrium, and absorbing states without pursuing abstract proofs.
- Examples
- Uses four-carrier market shares, a running-or-not-running engine, and loan-delinquency categories.
- What you can do after
- Apply it via NumPy to calculate one- and two-period state distributions and interpret longer-run behavior.
Read how Statistics Every Programmer Needs explains Markov chain →
Fabulous Adventures in Data Structures and Algorithms
A Markov process is a stochastic process in which the conditional distribution of the next state depends only on the current state.
- How it is used
- Models random state sequences for text generation and serves as the sequence structure used by the Metropolis algorithm.
- How deep it goes
- Defines the current-state rule and builds a generic sampler and count-based word model, then reuses the sampler for Metropolis transitions.
- Examples
- Uses a gambler’s chip-count walk and a word-pair generator trained on Shakespearean sentences.
- What you can do after
- Build it from scratch as a C# distribution of state sequences, with transitions that can also terminate a sequence.
Read how Fabulous Adventures in Data Structures and Algorithms explains Markov chain →
Machine Learning Algorithms in Depth
A Markov model is a stochastic model in which the conditional distribution of future states depends on the present state rather than the full history.
- How it is used
- Provides a transition-based model for page rank, posterior sampling, and latent-state time series.
- How deep it goes
- Defines first-order dependence and stochastic matrices, then develops state-distribution updates and long-term behavior; detailed sampling algorithms are deferred.
- Examples
- Uses a two-state transition diagram and a web-page link graph to illustrate transition probabilities.
- What you can do after
- Evaluate/understand it: form a discrete transition matrix, update state distributions, and interpret a stationary distribution in page rank or MCMC.
Read how Machine Learning Algorithms in Depth explains Markov chain →
Optimization Algorithms: AI techniques for design, planning, and control problems
A stochastic model representing a sequence of states in which each transition probability depends only on the current state.
- How it is used
- Models memoryless state transitions for stochastic optimization and other probabilistic processes.
- How deep it goes
- Defines the memoryless property and illustrates transitions, but does not develop formal probability theory.
- Examples
- Illustrates transitions among states A, B, and C and probabilistic acceptance of neighboring solutions during simulated annealing.
- What you can do after
- Apply it via SymPy's DiscreteMarkovChain for finite discrete-time homogeneous models.
Related concepts
- Markov decision processcontrasts with · A Markov decision process is a mathematical model of sequential decision-making defined by states, actions, probabilistic transitions, and rewards, with future dynamics depending on the current state and action.
- transition probabilityuses · A transition probability is the conditional probability that a stochastic process moves to a specified next state, possibly given its current state and an action.
- diffusion modelused by · A diffusion model is a generative model that learns to produce data by reversing a process that progressively adds noise to data.
- statehas part · State is information or a configuration describing the condition of a system, entity, or computation at a particular time and potentially influencing subsequent behavior or decisions.
- transition matrixuses · A transition matrix is a matrix whose entries represent the probabilities of moving between states in a stochastic process.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.