liveBook · Manning

Action-value function

As explained in Applied Reinforcement Learning

An action-value function maps a state-action pair to the expected return from taking the action and then following a policy.

An inventory manager needs to choose how much to order, not just judge whether current stock is sufficient. An action-value function, Qπ(s, a), estimates the expected return from taking action a in state s and then following policy π. Comparing action values helps turn a state assessment into a decision. section 4.3 section 6.3

The action value combines the immediate reward with the discounted value of what happens next. When transition probabilities and rewards are known, a one-step lookahead calculates Qπ(s, a) by averaging the immediate reward plus γVπ(s′) over possible next states s′. The discount factor γ controls how much future value counts. Unlike Qπ(s, a), the state-value function Vπ(s) evaluates the state under the policy without fixing the first action. section 4.2 section 4.4

Choosing an inventory order

Suppose inventory is empty. In one policy-improvement calculation, ordering two units has an immediate reward of −$40 and a next-state value estimate of $70. With a discount factor of 0.9, its action-value estimate is −$40 + 0.9 × $70 = $23. The alternatives are approximately $18 for ordering one unit and −$75 for ordering none, so the improved policy orders two. These are estimates for that iteration, not necessarily final values. section 4.3

Table 4.4 First iteration of policy improvement (with round up values)
State
Order 0
Order 1
Order 2
Best Action
New Policy π 1
0
$0-$0-$30 + 0.9×(-$50) = -$75
$0-$20-$30 + 0.9×$75 = $18
$0-$40-$0 + 0.9×$70 = $23
Order 2
π 1(0) = 2
1
$100-$0-$0 + 0.9×(-$50) = $55
$100-$20-$5 + 0.9×$75 = $143
$100-$40-$5 + 0.9×$70 = $118
Order 1
π 1(1) = 1
2
$100-$0-$5 + 0.9×$75 = $163
$100-$20-$10 + 0.9×$70 = $133
$100-$40-$10 + 0.9×$65 = $109
Order 0
π 1(2) = 0
+3
$100-$0-$10 + 0.9×$70 = $153
$100-$20-$15 + 0.9×$65 = $124
$100-$40-$15 + 0.9×$65 = $104
Order 0
π 1(+3) = 0

Compare the estimated values of three order quantities across inventory states and see which action the policy selects. section 4.3

Policy improvement uses this comparison in every state: calculate the value of each valid action and select the highest-valued one. In resource allocation, valid choices must respect constraints such as the available budget. Policy evaluation and improvement can then repeat until the recommended actions stop changing. section 4.4

A known model is not the only route to action values. Model-free methods learn Q(s, a) directly from experience, avoiding the need to know transition probabilities. A tabular approach keeps an estimate for each state-action pair, which gives the agent action-specific guidance but increases the number of values to track. section 6.3 section 6.7

SARSA updates an action value using the value of the actual next action selected by its policy. Q-learning instead uses the maximum next-action value, learning about a greedy policy while potentially behaving more exploratorily. Both use experience to improve estimates, but differ in which next action guides the update. section 6.4.1 section 6.7

The model-based approach depends on known transition probabilities and rewards, and large state spaces can make dynamic programming costly. Direct tabular learning avoids the model requirement but still needs an entry for every state-action pair. section 4.6 section 6.3

How Applied Reinforcement Learning covers it

How deep it goes
Develops Bellman one-step lookahead and greedy improvement, then explains direct tabular learning through temporal-difference updates.
Examples
Compares inventory order quantities and implements policy improvement for constrained project resource allocation.
What you can do after
Apply action-value comparisons to improve a policy and understand how tabular methods learn action values from experience.
How it connects
Connects action values to state-value functions, Bellman equations, generalized policy iteration, temporal-difference learning, SARSA, and Q-learning.

Also explained in

Compare how 6 books explain action-value function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.