liveBook · Manning

Setup-aware policy

As explained in Applied Reinforcement Learning

A setup-aware policy uses inventory thresholds and the current production setup to avoid unnecessary product switches while responding to critical shortages.

A setup-aware policy chooses what to produce using inventory levels and the current production setup. It favors staying with the current product to avoid costly switches, while allowing a switch when the other product is critically low. section 2.5

In the scooter-and-bike example, the policy checks conditions in order:

  • If the current product has fewer than five units in stock, keep producing it. section 2.5
  • Otherwise, if the other product has fewer than two units in stock, switch to producing it. section 2.5
  • Otherwise, idle production. section 2.5

Competing shortages

If the line is set up for scooters and inventory is four scooters and one bike, the policy produces scooters: the current-product safety check takes priority. With five scooters and one bike, it switches to bikes. section 2.5

Listing 2.8 Define heuristic policies
def greedy_policy(state, params: FactoryProblemDefinition):
    _day, inv_scooter, inv_bike, _setup = state #A
    if inv_scooter < inv_bike: #A
        return 0 #A
    elif inv_bike < inv_scooter: #A
        return 1 #A
    else:
        return 0 #A
 
 
def setup_aware_policy(state, params: FactoryProblemDefinition): #B
    _day, inv_scooter, inv_bike, current_setup = state
    safety_stock = 5   #C
    critical_stock = 2 #C
 
    if current_setup == 0 and inv_scooter < safety_stock: return 0 #D
    if current_setup == 1 and inv_bike < safety_stock: return 1 #D
 
    if current_setup == 1 and inv_scooter < critical_stock: return 0 #E
    if current_setup == 0 and inv_bike < critical_stock: return 1 #E
    return 2 #F
 
def always_produce_scooter_policy(state,
       params: FactoryProblemDefinition):
   return 0 #G
 
def always_produce_bike_policy(state,
       params: FactoryProblemDefinition):
   return 1 #H
 
def never_switch_policy(state, params: FactoryProblemDefinition): #I
    _day, inv_scooter, inv_bike, current_setup = state
 
    target_stock = 15 #J
 
    if current_setup == 0: return 0 if inv_scooter < target_stock else 2 #K
 
    else: return 1 if inv_bike < target_stock else 2 #L
 
def random_policy(state, params: FactoryProblemDefinition): #M
    return np.random.choice([0, 1, 2])

The policy listing shows the ordered threshold checks and compares them with other heuristic policies. section 2.5

A simulation can evaluate the policy day by day: it chooses an action, samples demand, updates the factory state, and tracks profit over a planning horizon. section 2.5

How Applied Reinforcement Learning covers it

How deep it goes
Specifies ordered safety and critical thresholds and an idle fallback, then places the policy in a simulation.
Examples
Uses scooter and bike inventories with safety stock of five and critical stock of two.
What you can do after
Evaluate the policy in a simulated factory by tracking daily and cumulative rewards.
How it connects
Connects the policy to Markov decision processes, stochastic demand, rewards, and production setup costs.

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.