liveBook · Manning

Markov decision process

As explained in Applied Reinforcement Learning

A Markov decision process is a formal model of sequential decision-making in which an agent selects actions in states, receives rewards, and causes probabilistic transitions whose relevant future behavior depends on the current state.

A business decision rarely happens in isolation. A factory’s production choice affects today’s profit, tomorrow’s inventory, and future choices. A Markov decision process (MDP) models such sequential decisions under uncertainty by connecting states, actions, rewards, and probabilistic transitions. section 2.5 section 2.7

An MDP describes the decision problem through four linked components:

  • State: the information needed to describe the current situation, such as inventory and machine setup. section 2.3 section 2.5
  • Action: a choice available in the current state, such as producing a product or leaving a machine idle. section 2.3 section 2.5
  • Transition: how the state can change after an action, including uncertainty such as fluctuating demand. section 2.3 section 2.5
  • Reward: feedback on the outcome, designed to reflect the business objective and relevant costs or penalties. section 2.7 section 2.5

The Markov property makes state design especially important: the current state must contain the information needed to predict what happens next, without relying on omitted history. For example, a pricing model that leaves out pending backorders may miss information relevant to future outcomes. section 2.7 section 2.4

Unlike a Markov chain, which describes probabilistic evolution without an agent’s control, an MDP includes actions that influence outcomes. A policy maps states to actions, weighing immediate rewards against future consequences. section 2.3 section 2.7

Production planning

A plant produces scooters and bikes over a fixed horizon. Its state records the day, each product’s inventory, and the current machine setup. Each day it can produce scooters, produce bikes, or idle. Demand is uncertain; reward reflects revenue and costs, including production, setup, holding, and backorders. Different policies can then be simulated and compared by cumulative profit. section 2.5

Listing 2.9 Simulation function
def simulate_trajectory(initial_state, policy_fn, 
                      params: FactoryProblemDefinition):
    
    state = initial_state
    daily_rewards = []
    cumulative_rewards = []
    total_profit = 0.0
 
    while state[0] < params.horizon: #A
        action = policy_fn(state, params) #B
        demand = sample_demand(params) #B
 
        next_state, prod_scooter, prod_bike, back_scooter,  #C
                back_bike = transition(state, action, demand, params) #C
 
        reward = reward_function(state, action, demand, 
                     params, prod_scooter, prod_bike, 
                           back_scooter, back_bike, next_state) #D
        daily_rewards.append(reward) #D
 
        total_profit += reward #E
        cumulative_rewards.append(total_profit) #E
 
        state = next_state #F
 
    return daily_rewards, cumulative_rewards, total_profit

Follow a simulation as it chooses actions, samples demand, updates the state, and accumulates rewards. section 2.5

When formulating an MDP, check that the reward encourages the real objective rather than an easy proxy. Balance penalties against other reward terms, and enforce hard constraints with action masking and penalties; soft constraints can be handled through penalties and trade-offs. section 2.7

How Applied Reinforcement Learning covers it

How deep it goes
Develops states, transitions, rewards, policies, and simulations, then implements policy iteration for resource allocation.
Examples
Uses dynamic pricing, factory production planning, and project resource allocation to map business variables into MDPs.
What you can do after
Build it from scratch by defining an MDP and simulating policies for a production-planning problem.
How it connects
Connects Markov chains, the Markov property, policies, reward engineering, and action masking.

Also explained in

Compare how 6 books explain Markov decision process →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.