liveBook · Manning

State design

As explained in Applied Reinforcement Learning

State design is the technique of selecting and structuring state information so that it captures the factors needed to predict relevant future outcomes and make decisions.

State design selects and structures the information an agent needs to predict relevant future outcomes and make decisions. In a Markov decision process, the state is the lens through which the agent views a changing business situation; omitting important history can make future outcomes depend on information the state does not contain. section 2.4 section 2.7

Use state design to translate an uncertain operational problem into a decision model:

  • Include the information that influences future transitions and decisions, such as inventory, demand forecasts, queues, staffing, or machine condition. section 2.4
  • Check for missing history: ignoring backorders, workload accumulation, maintenance timing, or operational history can break the Markov property. section 2.4
  • Connect the state to actions, rewards, transitions, objectives, and constraints so the model supports sequential decisions rather than isolated choices. section 2.7

Dynamic pricing

For daily online-store pricing, represent the state with inventory level, current price, demand forecast, and days remaining. Choose a price as the action and use daily profit as the reward. Omitting inventory or pending backorders leaves the agent unable to account for information that affects future outcomes. section 2.4

Table 2.1 Comparing Different Business Problems Modeled as Markov Decision Processes
Example
State design
Action design
Reward design
Non-Makrov if…
Dynamic Pricing in Online Store
Inventory level, current price, demand forecast, days left in month
Set daily price
Daily profit (revenue – costs)
Ignoring inventory level or pending backorders
Workforce Scheduling in Call Center
Queue length, available agents, forecasted call arrivals
Choose number of agents to schedule
Negative cost: waiting time penalties + salaries
Ignoring agent fatigue or cumulative workload history
Preventive Maintenance for Machines
Machine condition, time since last maintenance, environmental factors
Choose whether to maintain each machine
Cost of maintenance vs cost of breakdowns
Omitting time since last maintenance or operational history
Customer Retention Campaign
Active status, weeks since last engagement, recent activity, subscription tenure
Choose between offer, reminder, or silence
+1 for retention, 0 for churn
Missing recent activity history or engagement recency

Compare state, action, reward, and non-Markov design across dynamic pricing, workforce scheduling, preventive maintenance, and customer retention. section 2.4

How Applied Reinforcement Learning covers it

How deep it goes
Explains state selection through business examples and missing-history failures, but does not descend into algorithmic implementation.
Examples
Uses dynamic pricing, call-center scheduling, preventive maintenance, and customer retention as business modeling examples.
What it assumes
Assumes familiarity with business objectives and constraints while introducing states, actions, rewards, and transitions as a modeling structure.
What you can do after
Evaluate whether a business state contains the information needed to predict transitions and support decisions.
How it connects
Connects state design to Markov decision processes, Markov chains, policies, rewards, transitions, and constraints.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.