liveBook · Manning

Markov property

As explained in Applied Reinforcement Learning

The Markov property is the condition that the distribution of future states and rewards depends on the current state and action rather than on unrepresented earlier history.

A production planner needs to predict what happens after choosing what to make. The Markov property says that the current state and action contain the information needed to determine the distribution of the next state and reward; unrepresented earlier history adds nothing. This does not mean the future is certain: demand and other outcomes can still be random. section 2.5 section 2.7

The property is a claim about how the state is designed. A useful state summarizes the relevant effects of the past, so a decision model need not condition separately on every earlier event. Tracking an ever-growing history can make computation impractical, but leaving out important information can weaken predictions. section 2.2 section 2.4

A production-planning state

For a factory producing scooters and bikes, use the day, both inventory levels, and the current machine setup as the state. Inventory summarizes the effects of earlier production decisions and demand, while setup records an important current operating condition. The model treats this state as Markovian: given a production action, stochastic demand helps determine the next state. section 2.5

Look for missing information when designing a state:

  • In dynamic pricing, omitting inventory or pending backorders can leave future outcomes dependent on unrecorded history. section 2.4
  • In workforce scheduling, ignoring fatigue or cumulative workload can hide information relevant to future service performance. section 2.4
  • In preventive maintenance, omitting time since maintenance or operational history can leave relevant wear unrepresented. section 2.4

A Markov chain models probabilistic movement between states. A Markov decision process adds actions that influence those transitions. The assumption also lets a trajectory probability be written as a product of an initial-state probability, policy action probabilities, and current-state-and-action transition probabilities. section 2.7 section 9.2

Checking simulated customer transitions

Simulate customer states using a transition matrix, then compare the observed next-state distributions for customers currently in the same state but with different preceding states. Similar distributions are consistent with the property in this simulation; a finite simulation alone does not establish it for every real customer process. section 2.2

Listing 2.1 Simulate Markov Chain
def simulate_markov(P, initial_state, n_steps):
    n_states = P.shape[0]    #A
    chain = np.empty(n_steps, dtype=int)    #A
    chain[0] = initial_state    #B
    for t in range(1, n_steps):
        chain[t] = np.random.choice(n_states, p=P[chain[t-1]])    #C
    return chain

The simulation chooses each next customer state from the transition probabilities for the current state. section 2.2

How Applied Reinforcement Learning covers it

How deep it goes
Explains the assumption through conditional transitions and state-design failures, then demonstrates simulation and a production-planning model.
Examples
Uses customer-state transitions, pricing, workforce scheduling, maintenance, and a scooter-and-bike production model.
What it assumes
Assumes familiarity with Python, data structures, functions, classes, NumPy, and Pandas, with an essentials notebook available for review.
What you can do after
Evaluate state designs for omitted information that could make future outcomes depend on unrepresented history.
How it connects
Connects Markov chains, Markov decision processes, transition probabilities, policies, and policy gradients.

Also explained in

Compare how 4 books explain Markov property →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.