liveBook · Manning

Bellman equation

As explained in Reinforcement Learning from Human Feedback

A Bellman equation recursively relates a value or action-value function to immediate reward and the value of successor states.

The Bellman equation turns long-horizon reward estimation into a recursive update: estimate an action by combining its immediate reward with the discounted value of the successor state. This makes value estimation operational for reinforcement learning and temporal-difference updates. section 10.4 section 6.2.1

For a policy π, the general action-value form is Qπ(st, at) = E[rt + γVπ(st+1) | st, at]. Q measures expected future reward for a state–action pair, while V measures expected future reward from a state. In deterministic transitions, the expectation simplifies to Q(st, at) = rt + γV(st+1). section 6.2.1

The equation supports several practical RL operations:

  • Q-learning uses it to update estimates of how useful each state–action pair is as new experience arrives. section 10.4
  • Temporal-difference learning replaces a complete return with the discrepancy between a value prediction and the observed reward plus successor value. section 10.4 section 6.2.1
  • The same discrepancy defines the advantage estimate: A(st, at) = rt + γV(st+1) − V(st), measuring whether an action performs better than the state’s baseline. section 6.2.1

Interpreting a TD update

Suppose a value function predicts V(st) and an action produces reward rt before reaching st+1. The target rt + γV(st+1) is compared with V(st); their difference is the TD residual, which drives the value estimate toward what occurred. section 6.2.1

Equation 6.19
\[\label{eq:advantage_trick} A(s_t,a_t) = Q(s_t,a_t) - V(s_t) = r_t + \gamma V(s_{t+1}) - V(s_t).\]

Inspect the advantage identity connecting Q, V, immediate reward, successor value, and the TD residual. section 6.2.1

How Reinforcement Learning from Human Feedback covers it

How deep it goes
Defines general and deterministic Q-to-V recursions and derives the TD residual, but does not develop convergence proofs or full algorithms.
Examples
Uses Q-learning, deterministic language-model transitions, and the TD residual as concrete applications and interpretations.
What it assumes
Assumes states, actions, rewards, policies, discounting, and Markov decision processes; distinguishes value from action-value functions.
What you can do after
Apply it via Q-learning and temporal-difference updates to estimate state or action values from sequential experience.
How it connects
Connects Markov decision processes, value functions, action-value functions, temporal-difference learning, Q-learning, and advantage functions.

Also explained in

Compare how 3 books explain Bellman equation →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Reinforcement Learning from Human Feedback. The section links point to the parts of the book it draws on.