In a Markov decision process, the Bellman equation assigns each state its maximum expected cumulative reward. It compares each action’s immediate reward with the discounted expected value of possible next states. section 6.5 section 6.5.2
For each action, add its immediate reward to γ times the probability-weighted values of the possible next states; choose the action with the largest total. section 6.5.2
The discount factor γ controls how much future rewards matter. Repeating the recursive computation, for example through value iteration, can converge under appropriate conditions; the resulting values determine an optimal policy. section 6.5.2


