liveBook · Manning

Policy improvement

As explained in Applied Reinforcement Learning

Policy improvement is the process of deriving a policy that selects actions with higher estimated value than the current policy.

Policy improvement turns an estimate of how valuable different situations are into a better set of decisions. For an inventory manager, that means using current value estimates to reconsider how much to order at each stock level—not simply repeating the existing ordering rule. section 4.3

The method is part of generalized policy iteration. Policy evaluation estimates the value of following the current policy; policy improvement uses those estimates to choose better actions. The two steps repeat until the policy stops changing. section 4.3

A greedy improvement step works one state at a time:

  • Identify the actions that are valid in the state. section 4.4
  • For each action, look one step ahead: combine its immediate reward with the discounted value of each possible next state, weighted by transition probability. section 4.4
  • Choose the action with the highest expected value and record it in the new policy. Repeat for every state, then check whether any action changed. section 4.4

Improving an inventory rule

Suppose the initial rule orders one unit at every stock level. After an evaluation sweep, compare ordering zero, one, or two units. At zero stock, the estimated returns are −$75, $18, and $23 respectively, so the improved rule orders two units. At two units in stock, ordering nothing has the highest estimated return, $163. section 4.3

Table 4.4 First iteration of policy improvement (with round up values)
State
Order 0
Order 1
Order 2
Best Action
New Policy π 1
0
$0-$0-$30 + 0.9×(-$50) = -$75
$0-$20-$30 + 0.9×$75 = $18
$0-$40-$0 + 0.9×$70 = $23
Order 2
π 1(0) = 2
1
$100-$0-$0 + 0.9×(-$50) = $55
$100-$20-$5 + 0.9×$75 = $143
$100-$40-$5 + 0.9×$70 = $118
Order 1
π 1(1) = 1
2
$100-$0-$5 + 0.9×$75 = $163
$100-$20-$10 + 0.9×$70 = $133
$100-$40-$10 + 0.9×$65 = $109
Order 0
π 1(2) = 0
+3
$100-$0-$10 + 0.9×$70 = $153
$100-$20-$15 + 0.9×$65 = $124
$100-$40-$15 + 0.9×$65 = $104
Order 0
π 1(+3) = 0

Compare the estimated returns for all three ordering actions across inventory states. section 4.3

In a resource-allocation problem, the same logic compares feasible project allocations, accounting for remaining budget and possible project outcomes. With a fully evaluated policy and a known model, an unchanged greedy policy signals that the iteration can stop. Dynamic programming nevertheless depends on transition probabilities and rewards being known, and large state spaces can make it computationally expensive. section 4.4 section 4.6

How Applied Reinforcement Learning covers it

How deep it goes
Works through valid-action selection, one-step expected returns, and a stability check, alongside iterative policy evaluation.
Examples
Uses inventory ordering calculations and a project resource-allocation implementation to demonstrate improved decisions.
What you can do after
Build policy improvement from scratch by evaluating feasible actions and updating a policy until stable.
How it connects
Connects policy evaluation, Bellman equations, action-value functions, generalized policy iteration, and value iteration.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.