Action-value function
An action-value function maps a state-action pair to the expected cumulative return from taking that action and subsequently following a specified policy.
Grokking AI Algorithms, Second Edition
A Q-function maps a state-action pair to the expected cumulative reward obtained by taking the action and subsequently following a policy.
- How it is used
- Guides an agent’s action choice by scoring expected long-term rewards for actions in its current state.
- How deep it goes
- Works through table-based action scores and a neural-network training target, including discounted future value and squared-error loss.
- Examples
- Traces parking-lot states and actions, a Q-table, and a discounted target calculated from directional next-state scores.
- What you can do after
- Build it from scratch with a Python Q-table, or apply it via a neural-network approximator in deep Q-learning.
Read how Grokking AI Algorithms, Second Edition explains action-value function →
Optimization Algorithms: AI techniques for design, planning, and control problems
An action-value function estimates the expected cumulative future reward from taking a specified action in a state and then following a policy.
- How it is used
- Frames action selection as choosing actions with the greatest expected cumulative discounted reward.
- How deep it goes
- Defines Q(s,a), contrasts it with V(s), and works through one expected-return calculation without developing the full learning algorithm.
- Examples
- Uses the Reach the Treasure game and a probability-weighted trajectory whose expected return equals 5.728.
- What you can do after
- Evaluate the long-term desirability of taking a particular action from a particular state under a policy.
Evolutionary Deep Learning: Genetic algorithms and neural networks
A Q-function maps a state and action to an estimate of the expected cumulative reward from taking that action and following a policy thereafter.
- How it is used
- Deploys Q-values to balance exploration and exploitation while learning action choices through repeated environment interactions.
- How deep it goes
- Explains table representation and update mechanics, but deliberately stops short of a deeper Q-learning treatment.
- Examples
- Uses a Gym FrozenLake environment, a 16-by-4 Q-table, training episodes, and greedy simulated episodes.
- What you can do after
- Build it from scratch as a tabular Q-learning agent and evaluate its learned actions in an environment.
Build a Robo-Advisor with Python (From Scratch)
A Q-function maps a state and an action to the estimated cumulative reward obtained by taking that action and following a policy afterward.
- How it is used
- Deploys state-action estimates to learn asset-allocation policies for financial planning.
- How deep it goes
- Explains the update equation and implementation loop, stopping short of broader algorithmic variations.
- Examples
- Uses wealth-time-action arrays, random wealth transitions, terminal goals, exploratory action selection, and repeated Python episodes.
- What you can do after
- Builds a Q-learning loop that estimates action values and supports policy selection through exploration and exploitation.
Read how Build a Robo-Advisor with Python (From Scratch) explains action-value function →
Reinforcement Learning from Human Feedback
A Q-function maps a state-action pair to a scalar estimate of expected future reward.
- How it is used
- Uses state–action reward estimates to support Q-learning, temporal-difference updates, and policy-gradient learning signals.
- How deep it goes
- Defines Q, derives its Bellman relation to V, and connects it to advantages and TD residuals without developing a full Q-learning algorithm.
- Examples
- Uses deterministic language-model transitions, game-playing systems, adaptive control, and a policy-gradient TD-residual calculation.
- What you can do after
- Apply Q-values as Bellman targets, policy-gradient signals, or components of advantage and temporal-difference estimates.
Read how Reinforcement Learning from Human Feedback explains action-value function →
Related concepts
- Q-learningused by · Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.
- value functionkind of · A value function assigns or estimates the expected cumulative reward, utility, or quality associated with a state or action under a policy or decision rule.
- Bellman equationkind of · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
- Q-tablepart of · A Q-table is a lookup table storing estimated action values for state-action pairs in a tabular reinforcement-learning system.
- advantage functionused by · An advantage function in reinforcement learning estimates how much better or worse an action’s expected return is than a baseline value for the corresponding state or policy.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.