Action-value function

As explained in Grokking AI Algorithms, Second Edition

A Q-function maps a state-action pair to the expected cumulative reward obtained by taking the action and subsequently following a policy.

An action-value function, or Q-function, gives a state–action pair a score: the expected long-term reward from taking that action. Think of it as a scorecard for an agent deciding what to do next. In a Q-table, each row is a state, each column is an action, and each cell holds a Q-value. The agent can choose the highest-scoring action in its current row. section 10.10 section 10.6

Figure 10.11 An example Q-table and states that it represents

Compare the action scores for each illustrated state: the preferred move depends on where the agent is. section 10.6

The scores are learned, not known in advance. In Q-learning, the agent tries actions in an environment, receives rewards, and observes how its state changes. Early Q-values are guesses; repeated experience makes them more useful for choosing actions. The agent may also take a random action instead of following the table. section 10.6

A reward now, with a promising move later

Suppose the agent moves east, receives +1, and reaches a new state. Its predicted Q-values for the next state are 1.2 north, 0.5 south, 2.0 east, and 0.3 west. East has the highest predicted value. With a discount factor of 0.9, the training target for the move just taken is 1 + (0.9 × 2.0) = 2.8. The discount factor controls how much future rewards count alongside the immediate reward. section 10.11

For many possible states, deep Q-learning can use an artificial neural network instead of a table. The network predicts Q-values, then compares its prediction for the action taken with a target built from the reward and next-state predictions. The squared difference is its training loss. Whether stored in a table or predicted by a network, the practical question is the same: which action looks most rewarding from here? section 10.10 section 10.11

Figure 10.20 The difference between using a Q-table and an ANN for the Parking-Lot Problem

See how a Q-table and an artificial neural network represent learned action values differently in the parking-lot problem. section 10.10

How Grokking AI Algorithms, Second Edition covers it

How deep it goes

Works through table-based action scores and a neural-network training target, including discounted future value and squared-error loss.

Examples

Traces parking-lot states and actions, a Q-table, and a discounted target calculated from directional next-state scores.

What it assumes

Introduces Q-values through states, actions, and a scorecard analogy; the deep-learning extension uses artificial neural networks.

What you can do after

Build it from scratch with a Python Q-table, or apply it via a neural-network approximator in deep Q-learning.

How it connects

Connects Q-learning, reinforcement learning, discount factors, and deep Q-learning through action scores and reward-based learning.

liveBookmanning.com

Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsAn action-value function estimates the expected cumulative future reward from taking a specified action in a state and then following a policy.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA Q-function maps a state and action to an estimate of the expected cumulative reward from taking that action and following a policy thereafter.
• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)A Q-function maps a state and an action to the estimated cumulative reward obtained by taking that action and following a policy afterward.
• 

  liveBookmanning.com
  Reinforcement Learning from Human FeedbackA Q-function maps a state-action pair to a scalar estimate of expected future reward.

Compare how 5 books explain action-value function →

Related concepts

• Q-learningused by · Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.
• value functionkind of · A value function assigns or estimates the expected cumulative reward, utility, or quality associated with a state or action under a policy or decision rule.
• Bellman equationkind of · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
• Q-tablepart of · A Q-table is a lookup table storing estimated action values for state-action pairs in a tabular reinforcement-learning system.
• advantage functionused by · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.