When deciding how much inventory to order, immediate profit is only part of the story: today’s choice also affects tomorrow’s stock and costs. A value function estimates the expected cumulative return from a state or state-action pair, accounting for future rewards under a policy or optimal behavior. section 4.2 section 4.3
Two views support different decisions:
- The state-value function Vπ(s) measures how good it is to be in state s while following policy π. section 4.2
- The action-value function Qπ(s,a) measures the value of taking action a in state s and then following policy π. section 4.2 section 4.6
The Bellman equation connects a current value to immediate reward and discounted future value. For a state value, average over the actions the policy may choose and the possible next states, weighted by their probabilities. The discount factor γ controls how much future value contributes. section 4.2
Values guide policy improvement. First, evaluate the current policy by updating state values until changes are small. Then compare the available actions using those values and choose a better action in each state. Generalized policy iteration repeats evaluation and improvement until the policy stabilizes. section 4.3 section 4.4
Inventory ordering
Suppose a business earns $100 per unit sold and pays ordering, holding, and stockout costs. Starting with zero value estimates and a policy of ordering one unit, the first evaluation gives an empty-stock state a value of −$50. Using those estimates in the next improvement step, ordering two units has an estimated value of $23, versus $18 for ordering one. These are iteration estimates, not final long-run values. section 4.3
Table 4.4 First iteration of policy improvement (with round up values)
|
State
|
Order 0
|
Order 1
|
Order 2
|
Best Action
|
New Policy π
1
|
|
0
|
$0-$0-$30 + 0.9×(-$50) = -$75
|
$0-$20-$30 + 0.9×$75 = $18
|
$0-$40-$0 + 0.9×$70 = $23
|
Order 2
|
π
1(0) = 2
|
|
1
|
$100-$0-$0 + 0.9×(-$50) = $55
|
$100-$20-$5 + 0.9×$75 = $143
|
$100-$40-$5 + 0.9×$70 = $118
|
Order 1
|
π
1(1) = 1
|
|
2
|
$100-$0-$5 + 0.9×$75 = $163
|
$100-$20-$10 + 0.9×$70 = $133
|
$100-$40-$10 + 0.9×$65 = $109
|
Order 0
|
π
1(2) = 0
|
|
+3
|
$100-$0-$10 + 0.9×$70 = $153
|
$100-$20-$15 + 0.9×$65 = $124
|
$100-$40-$15 + 0.9×$65 = $104
|
Order 0
|
π
1(+3) = 0
|
Compare the immediate and discounted future components used to select an order quantity in each inventory state. section 4.3
Restaurant scheduling
A restaurant can seat a small party now or wait for a potentially more lucrative large party. With four hours left, one simulation learned a value of $85.17, compared with an analytical optimal value of $85.47. The values were closer in some states than others, so the comparison matters more than any single estimate. section 6.2
Value estimates can come from different methods. Dynamic programming computes them using known transition probabilities and rewards; temporal-difference learning updates them from experience. In deep reinforcement learning, a network can represent action values, as in DQN, or share a body with a policy, as in an actor–critic approach. A perfect model and large state spaces can limit dynamic programming in practice. section 4.6 section 6.2 section 12.1



