liveBook · Manning

Bellman equation

As explained in Timeless Algorithms, The Foundational Ideas

The Bellman equation is a recursive equation that expresses the value of a decision state in terms of immediate reward and the value of subsequent states.

In a Markov decision process, the Bellman equation assigns each state its maximum expected cumulative reward. It compares each action’s immediate reward with the discounted expected value of possible next states. section 6.5 section 6.5.2

V*(s)=maxa⁡{(R(s, a)+ γ∑s'P(s'| s, a) V*(s')})

For each action, add its immediate reward to γ times the probability-weighted values of the possible next states; choose the action with the largest total. section 6.5.2

The discount factor γ controls how much future rewards matter. Repeating the recursive computation, for example through value iteration, can converge under appropriate conditions; the resulting values determine an optimal policy. section 6.5.2

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Unpacks optimal state values, rewards, transition probabilities, discounting, and value iteration, with convergence qualified by appropriate conditions.
What it assumes
Defines Markov decision process components—states, actions, transition probabilities, and rewards—instead of assuming them as prior knowledge.
What you can do after
Understand how immediate rewards and discounted expected future values determine optimal actions and state values.
How it connects
Connects dynamic programming, Markov decision processes, value iteration, and optimal policies through recursive state values.

Also explained in

Compare how 3 books explain Bellman equation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.