liveBook · Manning

Value function

As explained in Applied Reinforcement Learning

A value function estimates the expected cumulative return from a state or state-action pair under a specified policy or optimal behavior.

When deciding how much inventory to order, immediate profit is only part of the story: today’s choice also affects tomorrow’s stock and costs. A value function estimates the expected cumulative return from a state or state-action pair, accounting for future rewards under a policy or optimal behavior. section 4.2 section 4.3

Two views support different decisions:

  • The state-value function Vπ(s) measures how good it is to be in state s while following policy π. section 4.2
  • The action-value function Qπ(s,a) measures the value of taking action a in state s and then following policy π. section 4.2 section 4.6

The Bellman equation connects a current value to immediate reward and discounted future value. For a state value, average over the actions the policy may choose and the possible next states, weighted by their probabilities. The discount factor γ controls how much future value contributes. section 4.2

Values guide policy improvement. First, evaluate the current policy by updating state values until changes are small. Then compare the available actions using those values and choose a better action in each state. Generalized policy iteration repeats evaluation and improvement until the policy stabilizes. section 4.3 section 4.4

Inventory ordering

Suppose a business earns $100 per unit sold and pays ordering, holding, and stockout costs. Starting with zero value estimates and a policy of ordering one unit, the first evaluation gives an empty-stock state a value of −$50. Using those estimates in the next improvement step, ordering two units has an estimated value of $23, versus $18 for ordering one. These are iteration estimates, not final long-run values. section 4.3

Table 4.4 First iteration of policy improvement (with round up values)
State
Order 0
Order 1
Order 2
Best Action
New Policy π 1
0
$0-$0-$30 + 0.9×(-$50) = -$75
$0-$20-$30 + 0.9×$75 = $18
$0-$40-$0 + 0.9×$70 = $23
Order 2
π 1(0) = 2
1
$100-$0-$0 + 0.9×(-$50) = $55
$100-$20-$5 + 0.9×$75 = $143
$100-$40-$5 + 0.9×$70 = $118
Order 1
π 1(1) = 1
2
$100-$0-$5 + 0.9×$75 = $163
$100-$20-$10 + 0.9×$70 = $133
$100-$40-$10 + 0.9×$65 = $109
Order 0
π 1(2) = 0
+3
$100-$0-$10 + 0.9×$70 = $153
$100-$20-$15 + 0.9×$65 = $124
$100-$40-$15 + 0.9×$65 = $104
Order 0
π 1(+3) = 0

Compare the immediate and discounted future components used to select an order quantity in each inventory state. section 4.3

Restaurant scheduling

A restaurant can seat a small party now or wait for a potentially more lucrative large party. With four hours left, one simulation learned a value of $85.17, compared with an analytical optimal value of $85.47. The values were closer in some states than others, so the comparison matters more than any single estimate. section 6.2

Value estimates can come from different methods. Dynamic programming computes them using known transition probabilities and rewards; temporal-difference learning updates them from experience. In deep reinforcement learning, a network can represent action values, as in DQN, or share a body with a policy, as in an actor–critic approach. A perfect model and large state spaces can limit dynamic programming in practice. section 4.6 section 6.2 section 12.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains Bellman averaging and iterative evaluation, then implements policy improvement in a resource allocation problem.
Examples
Works through inventory ordering, project resource allocation, and restaurant table scheduling.
What you can do after
Build value estimates from scratch through iterative policy evaluation and use them to compare actions.
How it connects
Connects value functions to Bellman equations, policy iteration, temporal-difference learning, and actor–critic methods.

Also explained in

Compare how 4 books explain value function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.