Policy improvement turns an estimate of how valuable different situations are into a better set of decisions. For an inventory manager, that means using current value estimates to reconsider how much to order at each stock level—not simply repeating the existing ordering rule. section 4.3
The method is part of generalized policy iteration. Policy evaluation estimates the value of following the current policy; policy improvement uses those estimates to choose better actions. The two steps repeat until the policy stops changing. section 4.3
A greedy improvement step works one state at a time:
- Identify the actions that are valid in the state. section 4.4
- For each action, look one step ahead: combine its immediate reward with the discounted value of each possible next state, weighted by transition probability. section 4.4
- Choose the action with the highest expected value and record it in the new policy. Repeat for every state, then check whether any action changed. section 4.4
Improving an inventory rule
Suppose the initial rule orders one unit at every stock level. After an evaluation sweep, compare ordering zero, one, or two units. At zero stock, the estimated returns are −$75, $18, and $23 respectively, so the improved rule orders two units. At two units in stock, ordering nothing has the highest estimated return, $163. section 4.3
Table 4.4 First iteration of policy improvement (with round up values)
|
State
|
Order 0
|
Order 1
|
Order 2
|
Best Action
|
New Policy π
1
|
|
0
|
$0-$0-$30 + 0.9×(-$50) = -$75
|
$0-$20-$30 + 0.9×$75 = $18
|
$0-$40-$0 + 0.9×$70 = $23
|
Order 2
|
π
1(0) = 2
|
|
1
|
$100-$0-$0 + 0.9×(-$50) = $55
|
$100-$20-$5 + 0.9×$75 = $143
|
$100-$40-$5 + 0.9×$70 = $118
|
Order 1
|
π
1(1) = 1
|
|
2
|
$100-$0-$5 + 0.9×$75 = $163
|
$100-$20-$10 + 0.9×$70 = $133
|
$100-$40-$10 + 0.9×$65 = $109
|
Order 0
|
π
1(2) = 0
|
|
+3
|
$100-$0-$10 + 0.9×$70 = $153
|
$100-$20-$15 + 0.9×$65 = $124
|
$100-$40-$15 + 0.9×$65 = $104
|
Order 0
|
π
1(+3) = 0
|
Compare the estimated returns for all three ordering actions across inventory states. section 4.3
In a resource-allocation problem, the same logic compares feasible project allocations, accounting for remaining budget and possible project outcomes. With a fully evaluated policy and a known model, an unchanged greedy policy signals that the iteration can stop. Dynamic programming nevertheless depends on transition probabilities and rewards being known, and large state spaces can make it computationally expensive. section 4.4 section 4.6
