liveBook · Manning

Markov process

As explained in Applied Reinforcement Learning

A Markov process is a stochastic process whose future behavior is conditionally independent of the past given its current state.

A process is Markovian when its future behavior depends on its current state, not on its earlier history. For a business process, the state must capture the information needed to predict what comes next. Designing that state well takes domain knowledge; statistical tests can help check whether earlier states still affect next-state probabilities. section 2.2

Checking what the state remembers

For a software-service process, compare the next-state distributions when the current state is Buy Pro but the preceding states differ. If those distributions differ, the current state may not capture enough relevant history. section 2.2

Figure 2.2 Conditional probability of dependency of next state to all previous states.

Use the figure to see how the next state’s dependence on earlier states relates to the Markov property. section 2.2

How Applied Reinforcement Learning covers it

How deep it goes
Defines the memoryless condition, discusses state design and statistical checks, and stops short of action-based decision models.
Examples
Uses software-service transitions and compares next-state distributions for Buy Pro after different histories.
What you can do after
Evaluate whether a state captures relevant history by checking whether earlier states change next-state probabilities.
How it connects
Connects states and transition probabilities to Markov chains and Markov decision processes.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.