Action-value function

As explained in Optimization Algorithms: AI techniques for design, planning, and control problems

An action-value function estimates the expected cumulative future reward from taking a specified action in a state and then following a policy.

An action-value function, written Q(s,a), estimates the expected cumulative future reward after taking action a in state s and then following policy π. It measures the long-term goodness of a specific action in a specific state. section 12.1.2

Use Q-values to distinguish action choices within reinforcement learning:

• Contrast Q(s,a) with the state-value function V(s), which evaluates a state under a policy rather than a particular action. section 12.1.2
• Interpret future rewards through the discount factor γ: values near zero emphasize immediate rewards, while values near one consider more distant rewards. section 12.1.2
• Place action-value estimates alongside value iteration, policy iteration, and policy gradients as methods for learning policies that maximize expected cumulative discounted reward. section 12.1.2

A trajectory after moving right

A hand-worked trajectory combines possible rewards and transition probabilities to obtain an expected return of 5.728: 0.4 × 10 + 0.6 × 4 + 0.4 × 0.2 × (−10) + 0.4 × 0.8 × 4 × 0.1 = 5.728, with additional zero-reward terms shown in the full calculation. section 12.1.2

Figure 12.4 Reach the Treasure game

Use the Reach the Treasure game to visualize how an agent’s action choices connect to expected returns. section 12.1.2

How Optimization Algorithms: AI techniques for design, planning, and control problems covers it

How deep it goes

Defines Q(s,a), contrasts it with V(s), and works through one expected-return calculation without developing the full learning algorithm.

Examples

Uses the Reach the Treasure game and a probability-weighted trajectory whose expected return equals 5.728.

What you can do after

Evaluate the long-term desirability of taking a particular action from a particular state under a policy.

How it connects

Connects action-value functions to reinforcement learning, policies, state-value functions, discount factors, value iteration, policy iteration, and policy gradients.

liveBookmanning.com

Read Optimization Algorithms: AI techniques for design, planning, and control problems on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionA Q-function maps a state-action pair to the expected cumulative reward obtained by taking the action and subsequently following a policy.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA Q-function maps a state and action to an estimate of the expected cumulative reward from taking that action and following a policy thereafter.
• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)A Q-function maps a state and an action to the estimated cumulative reward obtained by taking that action and following a policy afterward.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA Q-function maps a state-action pair to a scalar estimate of expected future reward.

Compare how 5 books explain action-value function →

Related concepts

• Q-learningused by · Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.
• value functionkind of · A value function assigns or estimates the expected cumulative reward, utility, or quality associated with a state or action under a policy or decision rule.
• Bellman equationkind of · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
• Q-tablepart of · A Q-table is a lookup table storing estimated action values for state-action pairs in a tabular reinforcement-learning system.
• advantage functionused by · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.

This explainer was generated by Manning's knowledge graph from the text of Optimization Algorithms: AI techniques for design, planning, and control problems. The section links point to the parts of the book it draws on.