liveBook · Manning

Bellman equation

As explained in Applied Reinforcement Learning

A Bellman equation expresses the value of a state or state-action pair as an immediate reward plus the appropriately discounted value of future states.

An inventory decision can look costly today but prevent shortages tomorrow. The Bellman equation captures that trade-off: the long-term value of a state or state-action pair is the expected immediate reward plus the discounted value of what comes next. Each value depends recursively on future values, like one domino influencing the next. section 4.2 section 4.3

For a policy π, the state-value form is Vπ(s) = Eπ[R + γVπ(S′) | S = s]. Here R is the immediate reward, S′ is the next state, and γ discounts its future value. The expectation averages over actions selected by the policy and possible next states. The action-value form instead starts with a particular action in the current state. section 4.2 section 4.6

Table 4.1 Core elements of the Bellman equation
Element
Symbol
Description
How it is obtained
State
s
A representation of the environment at a specific point in time. It includes all the information the agent needs.
Defined by the environment; observed directly during interaction or simulation.
Action
a
A choice available to the agent in state s. Determines how the agent interacts with the environment.
Selected by the agent, either via a policy or during exploration.
Next State
s′
The resulting state after action a is taken in state s.
Observed from the environment after the action is executed.
Transition Probability
P(s′ | s, a)
The probability that action aa in state ss will lead to next state s′.
Estimated from data through sampling or modeled explicitly if dynamics are known.
Reward Function
R(s, a, s′)
The immediate reward received after transitioning from s to s' via action a.
Defined by the problem designer or observed as feedback from the environment.
Policy
π(a | s)
The agent’s behavior strategy: probability of taking action a in state s.
Learned through training, or predefined in a rule-based or heuristic form.
Discount Factor
γ ∈ [0,1]
A number balancing the importance of immediate vs. future rewards.
Chosen by the designer based on how far into the future the agent should plan.

Use the table to connect states, actions, transition probabilities, and rewards to a decision problem. section 4.2

An inventory decision

With no stock, ordering two units gives an immediate reward of −$40 in one calculation. If the next state's estimated value is $70 and γ is 0.9, the action's estimated return is −$40 + 0.9 × $70 = $23. Ordering one unit has an estimated return of $18, so the improvement step chooses two. section 4.3

Table 4.4 First iteration of policy improvement (with round up values)
State
Order 0
Order 1
Order 2
Best Action
New Policy π 1
0
$0-$0-$30 + 0.9×(-$50) = -$75
$0-$20-$30 + 0.9×$75 = $18
$0-$40-$0 + 0.9×$70 = $23
Order 2
π 1(0) = 2
1
$100-$0-$0 + 0.9×(-$50) = $55
$100-$20-$5 + 0.9×$75 = $143
$100-$40-$5 + 0.9×$70 = $118
Order 1
π 1(1) = 1
2
$100-$0-$5 + 0.9×$75 = $163
$100-$20-$10 + 0.9×$70 = $133
$100-$40-$10 + 0.9×$65 = $109
Order 0
π 1(2) = 0
+3
$100-$0-$10 + 0.9×$70 = $153
$100-$20-$15 + 0.9×$65 = $124
$100-$40-$15 + 0.9×$65 = $104
Order 0
π 1(+3) = 0

Compare the candidate orders and their immediate and discounted future values across inventory states. section 4.3

Generalized policy iteration puts the equation to work in a loop. Policy evaluation repeatedly updates values under the current policy, averaging possible outcomes by their probabilities. Policy improvement then compares actions using those values and selects better ones. The cycle repeats until the policy stops changing. section 4.3 section 4.4

Dynamic programming uses this recursion when transition probabilities and rewards are known. Its practical limits are the need for a complete model and the computational and memory cost of large state spaces; those are limitations of the dynamic programming approach, not of the equation's intuition. section 4.6 section 4.1

How Applied Reinforcement Learning covers it

How deep it goes
Explains state and action values, iterative policy evaluation, and greedy improvement, with calculations and implementation.
Examples
Works through inventory ordering calculations and a project resource-allocation problem.
What you can do after
Build it from scratch to evaluate and improve a policy for a modeled resource-allocation problem.
How it connects
Connects the equation to value functions, Markov decision processes, dynamic programming, policy iteration, and value iteration.

Also explained in

Compare how 3 books explain Bellman equation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.