A Markov chain models how a system moves probabilistically between states. It describes prediction: transitions unfold according to chance, rather than actions chosen to steer the system. section 2.1 section 2.3
The Markov property provides a shortcut for predicting the next state without tracking every earlier state. Considering long histories can make computation grow impractically complex. section 2.2
Figure 2.3 Software service transitions graph and its matrix representation.
A software-service example represents customer-state transitions as both a graph and a matrix, whose rows sum to one. section 2.2
Simulating customer states
Starting from an initial customer state, sample successive states using the transition probabilities. Then count observed next states to estimate their distribution, including transitions from a state such as “Buy Pro.” section 2.2
A Markov chain describes what may happen without intervention. A Markov decision process adds choices that influence transitions and rewards, making it suitable for optimizing outcomes. section 2.3






