In production planning, a greedy policy is a simple rule for deciding what to make today: compare current scooter and bike stocks and produce whichever is lower. If stocks are equal, produce scooters. The rule does not consider the current production setup or switching costs. section 2.5
Choosing a product
If scooter stock is 3 and bike stock is 5, produce scooters. If the stocks are both 5, the tie also goes to scooters. section 2.5
Listing 2.8 Define heuristic policies
def greedy_policy(state, params: FactoryProblemDefinition):
_day, inv_scooter, inv_bike, _setup = state #A
if inv_scooter < inv_bike: #A
return 0 #A
elif inv_bike < inv_scooter: #A
return 1 #A
else:
return 0 #A
def setup_aware_policy(state, params: FactoryProblemDefinition): #B
_day, inv_scooter, inv_bike, current_setup = state
safety_stock = 5 #C
critical_stock = 2 #C
if current_setup == 0 and inv_scooter < safety_stock: return 0 #D
if current_setup == 1 and inv_bike < safety_stock: return 1 #D
if current_setup == 1 and inv_scooter < critical_stock: return 0 #E
if current_setup == 0 and inv_bike < critical_stock: return 1 #E
return 2 #F
def always_produce_scooter_policy(state,
params: FactoryProblemDefinition):
return 0 #G
def always_produce_bike_policy(state,
params: FactoryProblemDefinition):
return 1 #H
def never_switch_policy(state, params: FactoryProblemDefinition): #I
_day, inv_scooter, inv_bike, current_setup = state
target_stock = 15 #J
if current_setup == 0: return 0 if inv_scooter < target_stock else 2 #K
else: return 1 if inv_bike < target_stock else 2 #L
def random_policy(state, params: FactoryProblemDefinition): #M
return np.random.choice([0, 1, 2])
Compare the stock-only greedy rule with the setup-aware policy and its inventory thresholds. section 2.5
Greedy also describes a different decision rule in generalized policy iteration. There, the policy chooses the action with the highest expected immediate reward plus discounted value of successor states. Policy evaluation estimates values; policy improvement then chooses greedily using those estimates. Unlike the factory rule, this choice explicitly considers estimated future value. section 2.5 section 4.3
Improving an inventory policy
For an electronics retailer deciding weekly order quantities, one policy-improvement calculation selects ordering two units when stock is zero: its estimated value is $23, compared with $18 for ordering one and −$75 for ordering none. section 4.3
Table 4.4 First iteration of policy improvement (with round up values)
|
State
|
Order 0
|
Order 1
|
Order 2
|
Best Action
|
New Policy π
1
|
|
0
|
$0-$0-$30 + 0.9×(-$50) = -$75
|
$0-$20-$30 + 0.9×$75 = $18
|
$0-$40-$0 + 0.9×$70 = $23
|
Order 2
|
π
1(0) = 2
|
|
1
|
$100-$0-$0 + 0.9×(-$50) = $55
|
$100-$20-$5 + 0.9×$75 = $143
|
$100-$40-$5 + 0.9×$70 = $118
|
Order 1
|
π
1(1) = 1
|
|
2
|
$100-$0-$5 + 0.9×$75 = $163
|
$100-$20-$10 + 0.9×$70 = $133
|
$100-$40-$10 + 0.9×$65 = $109
|
Order 0
|
π
1(2) = 0
|
|
+3
|
$100-$0-$10 + 0.9×$70 = $153
|
$100-$20-$15 + 0.9×$65 = $124
|
$100-$40-$15 + 0.9×$65 = $104
|
Order 0
|
π
1(+3) = 0
|
Inspect the first policy-improvement choices across inventory levels. section 4.3
