liveBook · Manning

Greedy policy

As explained in Applied Reinforcement Learning

A greedy policy selects the product with the lower current stock level, resolving equal stock levels by choosing scooter.

In production planning, a greedy policy is a simple rule for deciding what to make today: compare current scooter and bike stocks and produce whichever is lower. If stocks are equal, produce scooters. The rule does not consider the current production setup or switching costs. section 2.5

Choosing a product

If scooter stock is 3 and bike stock is 5, produce scooters. If the stocks are both 5, the tie also goes to scooters. section 2.5

Listing 2.8 Define heuristic policies
def greedy_policy(state, params: FactoryProblemDefinition):
    _day, inv_scooter, inv_bike, _setup = state #A
    if inv_scooter < inv_bike: #A
        return 0 #A
    elif inv_bike < inv_scooter: #A
        return 1 #A
    else:
        return 0 #A
 
 
def setup_aware_policy(state, params: FactoryProblemDefinition): #B
    _day, inv_scooter, inv_bike, current_setup = state
    safety_stock = 5   #C
    critical_stock = 2 #C
 
    if current_setup == 0 and inv_scooter < safety_stock: return 0 #D
    if current_setup == 1 and inv_bike < safety_stock: return 1 #D
 
    if current_setup == 1 and inv_scooter < critical_stock: return 0 #E
    if current_setup == 0 and inv_bike < critical_stock: return 1 #E
    return 2 #F
 
def always_produce_scooter_policy(state,
       params: FactoryProblemDefinition):
   return 0 #G
 
def always_produce_bike_policy(state,
       params: FactoryProblemDefinition):
   return 1 #H
 
def never_switch_policy(state, params: FactoryProblemDefinition): #I
    _day, inv_scooter, inv_bike, current_setup = state
 
    target_stock = 15 #J
 
    if current_setup == 0: return 0 if inv_scooter < target_stock else 2 #K
 
    else: return 1 if inv_bike < target_stock else 2 #L
 
def random_policy(state, params: FactoryProblemDefinition): #M
    return np.random.choice([0, 1, 2])

Compare the stock-only greedy rule with the setup-aware policy and its inventory thresholds. section 2.5

Greedy also describes a different decision rule in generalized policy iteration. There, the policy chooses the action with the highest expected immediate reward plus discounted value of successor states. Policy evaluation estimates values; policy improvement then chooses greedily using those estimates. Unlike the factory rule, this choice explicitly considers estimated future value. section 2.5 section 4.3

Improving an inventory policy

For an electronics retailer deciding weekly order quantities, one policy-improvement calculation selects ordering two units when stock is zero: its estimated value is $23, compared with $18 for ordering one and −$75 for ordering none. section 4.3

Table 4.4 First iteration of policy improvement (with round up values)
State
Order 0
Order 1
Order 2
Best Action
New Policy π 1
0
$0-$0-$30 + 0.9×(-$50) = -$75
$0-$20-$30 + 0.9×$75 = $18
$0-$40-$0 + 0.9×$70 = $23
Order 2
π 1(0) = 2
1
$100-$0-$0 + 0.9×(-$50) = $55
$100-$20-$5 + 0.9×$75 = $143
$100-$40-$5 + 0.9×$70 = $118
Order 1
π 1(1) = 1
2
$100-$0-$5 + 0.9×$75 = $163
$100-$20-$10 + 0.9×$70 = $133
$100-$40-$10 + 0.9×$65 = $109
Order 0
π 1(2) = 0
+3
$100-$0-$10 + 0.9×$70 = $153
$100-$20-$15 + 0.9×$65 = $124
$100-$40-$15 + 0.9×$65 = $104
Order 0
π 1(+3) = 0

Inspect the first policy-improvement choices across inventory levels. section 4.3

How Applied Reinforcement Learning covers it

How deep it goes
Defines the factory decision rule and works through greedy improvement using estimated rewards and successor-state values.
Examples
Uses scooter and bike production and a retailer’s weekly inventory-ordering decisions.
What you can do after
Evaluate and understand stock-based decisions and greedy policy improvement from estimated values.
How it connects
Connects inventory management with production setup, switching costs, policy evaluation, value functions, and generalized policy iteration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.