State design selects and structures the information an agent needs to predict relevant future outcomes and make decisions. In a Markov decision process, the state is the lens through which the agent views a changing business situation; omitting important history can make future outcomes depend on information the state does not contain. section 2.4 section 2.7
Use state design to translate an uncertain operational problem into a decision model:
- Include the information that influences future transitions and decisions, such as inventory, demand forecasts, queues, staffing, or machine condition. section 2.4
- Check for missing history: ignoring backorders, workload accumulation, maintenance timing, or operational history can break the Markov property. section 2.4
- Connect the state to actions, rewards, transitions, objectives, and constraints so the model supports sequential decisions rather than isolated choices. section 2.7
Dynamic pricing
For daily online-store pricing, represent the state with inventory level, current price, demand forecast, and days remaining. Choose a price as the action and use daily profit as the reward. Omitting inventory or pending backorders leaves the agent unable to account for information that affects future outcomes. section 2.4
Table 2.1 Comparing Different Business Problems Modeled as Markov Decision Processes
|
Example
|
State design
|
Action design
|
Reward design
|
Non-Makrov if…
|
|
Dynamic Pricing in Online Store
|
Inventory level, current price, demand forecast, days left in month
|
Set daily price
|
Daily profit (revenue – costs)
|
Ignoring inventory level or pending backorders
|
|
Workforce Scheduling in Call Center
|
Queue length, available agents, forecasted call arrivals
|
Choose number of agents to schedule
|
Negative cost: waiting time penalties + salaries
|
Ignoring agent fatigue or cumulative workload history
|
|
Preventive Maintenance for Machines
|
Machine condition, time since last maintenance, environmental factors
|
Choose whether to maintain each machine
|
Cost of maintenance vs cost of breakdowns
|
Omitting time since last maintenance or operational history
|
|
Customer Retention Campaign
|
Active status, weeks since last engagement, recent activity, subscription tenure
|
Choose between offer, reminder, or silence
|
+1 for retention, 0 for churn
|
Missing recent activity history or engagement recency
|
Compare state, action, reward, and non-Markov design across dynamic pricing, workforce scheduling, preventive maintenance, and customer retention. section 2.4
