liveBook · Manning

Value iteration

As explained in Optimization Algorithms: AI techniques for design, planning, and control problems

Value iteration is an iterative reinforcement-learning or dynamic-programming method that updates state values toward optimal expected returns.

Value iteration is an iterative reinforcement-learning and dynamic-programming method for moving state values toward an optimal policy. It belongs with policy iteration and policy gradients as a way to maximize expected cumulative discounted reward. section 12.1.2

The surrounding framework defines state-value V(s) as expected future reward from a state and action-value Q(s,a) as expected future reward for a state-action pair. Discounting controls how strongly future rewards influence those values. section 12.1.2

Reach the Treasure

A treasure-seeking agent chooses between moving left and right from observations. One illustrated trajectory calculates an expected return of 5.728 by combining action probabilities, rewards, and later transitions. section 12.1.2

How Optimization Algorithms: AI techniques for design, planning, and control problems covers it

How deep it goes
Defines its reinforcement-learning role and related value functions but does not provide update steps or implementation details.
Examples
Uses the Reach the Treasure game, an agent-environment figure, and a worked expected-return calculation yielding 5.728.
What it assumes
Assumes familiarity with states, actions, policies, rewards, trajectories, and expected returns while defining value functions.
What you can do after
Evaluate how state values, action values, policies, and discounting support maximizing expected cumulative reward.
How it connects
Connects value iteration with reinforcement learning, policy iteration, policy gradients, value functions, and discounted rewards.
Cover of Optimization Algorithms: AI techniques for design, planning, and control problems
liveBookmanning.com
Read Optimization Algorithms: AI techniques for design, planning, and control problems on liveBook Get the book at manning.com

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Optimization Algorithms: AI techniques for design, planning, and control problems. The section links point to the parts of the book it draws on.