liveBook · Manning

Markov chain

As explained in Applied Reinforcement Learning

A Markov chain is a stochastic model describing how a system evolves among states when the next state depends probabilistically on the current state.

A Markov chain models how a system moves probabilistically between states. It describes prediction: transitions unfold according to chance, rather than actions chosen to steer the system. section 2.1 section 2.3

The Markov property provides a shortcut for predicting the next state without tracking every earlier state. Considering long histories can make computation grow impractically complex. section 2.2

Figure 2.3 Software service transitions graph and its matrix representation.

A software-service example represents customer-state transitions as both a graph and a matrix, whose rows sum to one. section 2.2

Simulating customer states

Starting from an initial customer state, sample successive states using the transition probabilities. Then count observed next states to estimate their distribution, including transitions from a state such as “Buy Pro.” section 2.2

A Markov chain describes what may happen without intervention. A Markov decision process adds choices that influence transitions and rewards, making it suitable for optimizing outcomes. section 2.3

How Applied Reinforcement Learning covers it

How deep it goes
Explains the Markov-property shortcut and demonstrates simulation and empirical transition estimates, without treating the chain as an optimization model.
Examples
Uses software-service customer states, a transition graph and matrix, and simulated next-state frequencies.
What it assumes
Assumes preparation in Python, data structures, functions, NumPy, and Pandas rather than establishing those foundations here.
What you can do after
Apply it via NumPy to simulate state transitions and estimate next-state distributions from observed frequencies.
How it connects
Links Markov chains to the Markov property, transition matrices, and Markov decision processes.

Also explained in

Compare how 7 books explain Markov chain →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.