A production planner needs to predict what happens after choosing what to make. The Markov property says that the current state and action contain the information needed to determine the distribution of the next state and reward; unrepresented earlier history adds nothing. This does not mean the future is certain: demand and other outcomes can still be random. section 2.5 section 2.7
The property is a claim about how the state is designed. A useful state summarizes the relevant effects of the past, so a decision model need not condition separately on every earlier event. Tracking an ever-growing history can make computation impractical, but leaving out important information can weaken predictions. section 2.2 section 2.4
A production-planning state
For a factory producing scooters and bikes, use the day, both inventory levels, and the current machine setup as the state. Inventory summarizes the effects of earlier production decisions and demand, while setup records an important current operating condition. The model treats this state as Markovian: given a production action, stochastic demand helps determine the next state. section 2.5
Look for missing information when designing a state:
- In dynamic pricing, omitting inventory or pending backorders can leave future outcomes dependent on unrecorded history. section 2.4
- In workforce scheduling, ignoring fatigue or cumulative workload can hide information relevant to future service performance. section 2.4
- In preventive maintenance, omitting time since maintenance or operational history can leave relevant wear unrepresented. section 2.4
A Markov chain models probabilistic movement between states. A Markov decision process adds actions that influence those transitions. The assumption also lets a trajectory probability be written as a product of an initial-state probability, policy action probabilities, and current-state-and-action transition probabilities. section 2.7 section 9.2
Checking simulated customer transitions
Simulate customer states using a transition matrix, then compare the observed next-state distributions for customers currently in the same state but with different preceding states. Similar distributions are consistent with the property in this simulation; a finite simulation alone does not establish it for every real customer process. section 2.2
Listing 2.1 Simulate Markov Chain
def simulate_markov(P, initial_state, n_steps):
n_states = P.shape[0] #A
chain = np.empty(n_steps, dtype=int) #A
chain[0] = initial_state #B
for t in range(1, n_steps):
chain[t] = np.random.choice(n_states, p=P[chain[t-1]]) #C
return chain
The simulation chooses each next customer state from the transition probabilities for the current state. section 2.2



