An inventory manager needs to choose how much to order, not just judge whether current stock is sufficient. An action-value function, Qπ(s, a), estimates the expected return from taking action a in state s and then following policy π. Comparing action values helps turn a state assessment into a decision. section 4.3 section 6.3
The action value combines the immediate reward with the discounted value of what happens next. When transition probabilities and rewards are known, a one-step lookahead calculates Qπ(s, a) by averaging the immediate reward plus γVπ(s′) over possible next states s′. The discount factor γ controls how much future value counts. Unlike Qπ(s, a), the state-value function Vπ(s) evaluates the state under the policy without fixing the first action. section 4.2 section 4.4
Choosing an inventory order
Suppose inventory is empty. In one policy-improvement calculation, ordering two units has an immediate reward of −$40 and a next-state value estimate of $70. With a discount factor of 0.9, its action-value estimate is −$40 + 0.9 × $70 = $23. The alternatives are approximately $18 for ordering one unit and −$75 for ordering none, so the improved policy orders two. These are estimates for that iteration, not necessarily final values. section 4.3
Table 4.4 First iteration of policy improvement (with round up values)
|
State
|
Order 0
|
Order 1
|
Order 2
|
Best Action
|
New Policy π
1
|
|
0
|
$0-$0-$30 + 0.9×(-$50) = -$75
|
$0-$20-$30 + 0.9×$75 = $18
|
$0-$40-$0 + 0.9×$70 = $23
|
Order 2
|
π
1(0) = 2
|
|
1
|
$100-$0-$0 + 0.9×(-$50) = $55
|
$100-$20-$5 + 0.9×$75 = $143
|
$100-$40-$5 + 0.9×$70 = $118
|
Order 1
|
π
1(1) = 1
|
|
2
|
$100-$0-$5 + 0.9×$75 = $163
|
$100-$20-$10 + 0.9×$70 = $133
|
$100-$40-$10 + 0.9×$65 = $109
|
Order 0
|
π
1(2) = 0
|
|
+3
|
$100-$0-$10 + 0.9×$70 = $153
|
$100-$20-$15 + 0.9×$65 = $124
|
$100-$40-$15 + 0.9×$65 = $104
|
Order 0
|
π
1(+3) = 0
|
Compare the estimated values of three order quantities across inventory states and see which action the policy selects. section 4.3
Policy improvement uses this comparison in every state: calculate the value of each valid action and select the highest-valued one. In resource allocation, valid choices must respect constraints such as the available budget. Policy evaluation and improvement can then repeat until the recommended actions stop changing. section 4.4
A known model is not the only route to action values. Model-free methods learn Q(s, a) directly from experience, avoiding the need to know transition probabilities. A tabular approach keeps an estimate for each state-action pair, which gives the agent action-specific guidance but increases the number of values to track. section 6.3 section 6.7
SARSA updates an action value using the value of the actual next action selected by its policy. Q-learning instead uses the maximum next-action value, learning about a greedy policy while potentially behaving more exploratorily. Both use experience to improve estimates, but differ in which next action guides the update. section 6.4.1 section 6.7
The model-based approach depends on known transition probabilities and rewards, and large state spaces can make dynamic programming costly. Direct tabular learning avoids the model requirement but still needs an entry for every state-action pair. section 4.6 section 6.3





