liveBook · Manning

Action-value function

Also called q-function, q-value

An action-value function maps a state and action to the expected cumulative return from taking that action and then following a specified policy.

How 6 Manning books explain it

Applied Reinforcement Learning

An action-value function maps a state-action pair to the expected return from taking the action and then following a policy.

How it is used
Uses action values to compare decisions during policy improvement and to select actions without a model.
How deep it goes
Develops Bellman one-step lookahead and greedy improvement, then explains direct tabular learning through temporal-difference updates.
Examples
Compares inventory order quantities and implements policy improvement for constrained project resource allocation.
What you can do after
Apply action-value comparisons to improve a policy and understand how tabular methods learn action values from experience.

Read how Applied Reinforcement Learning explains action-value function →

Grokking AI Algorithms, Second Edition

A Q-function maps a state-action pair to the expected cumulative reward obtained by taking the action and subsequently following a policy.

How it is used
Guides action selection toward cumulative rewards in Q-learning and deep Q-learning.
How deep it goes
Explains table-based scores and neural-network approximation, including a worked target and squared loss.
Examples
Uses parking-lot states and actions, plus an eastward move with four predicted next-state values.
What you can do after
Evaluate Q-values to choose a high-scoring action and calculate a neural-network training target from a transition.

Read how Grokking AI Algorithms, Second Edition explains action-value function →

Optimization Algorithms: AI techniques for design, planning, and control problems

An action-value function estimates the expected cumulative future reward from taking a specified action in a state and then following a policy.

How it is used
Frames action selection as choosing actions with the greatest expected cumulative discounted reward.
How deep it goes
Defines Q(s,a), contrasts it with V(s), and works through one expected-return calculation without developing the full learning algorithm.
Examples
Uses the Reach the Treasure game and a probability-weighted trajectory whose expected return equals 5.728.
What you can do after
Evaluate the long-term desirability of taking a particular action from a particular state under a policy.

Read how Optimization Algorithms: AI techniques for design, planning, and control problems explains action-value function →

Evolutionary Deep Learning: Genetic algorithms and neural networks

A Q-function maps a state and action to an estimate of the expected cumulative reward from taking that action and following a policy thereafter.

How it is used
Deploys Q-values to balance exploration and exploitation while learning action choices through repeated environment interactions.
How deep it goes
Explains table representation and update mechanics, but deliberately stops short of a deeper Q-learning treatment.
Examples
Uses a Gym FrozenLake environment, a 16-by-4 Q-table, training episodes, and greedy simulated episodes.
What you can do after
Build it from scratch as a tabular Q-learning agent and evaluate its learned actions in an environment.

Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains action-value function →

Build a Robo-Advisor with Python (From Scratch)

A Q-function maps a state and an action to the estimated cumulative reward obtained by taking that action and following a policy afterward.

How it is used
Deploys state-action estimates to learn asset-allocation policies for financial planning.
How deep it goes
Explains the update equation and implementation loop, stopping short of broader algorithmic variations.
Examples
Uses wealth-time-action arrays, random wealth transitions, terminal goals, exploratory action selection, and repeated Python episodes.
What you can do after
Builds a Q-learning loop that estimates action values and supports policy selection through exploration and exploitation.

Read how Build a Robo-Advisor with Python (From Scratch) explains action-value function →

Reinforcement Learning from Human Feedback

A Q-function maps a state-action pair to a scalar estimate of expected future reward.

How it is used
Uses state–action reward estimates to support Q-learning, temporal-difference updates, and policy-gradient learning signals.
How deep it goes
Defines Q, derives its Bellman relation to V, and connects it to advantages and TD residuals without developing a full Q-learning algorithm.
Examples
Uses deterministic language-model transitions, game-playing systems, adaptive control, and a policy-gradient TD-residual calculation.
What you can do after
Apply Q-values as Bellman targets, policy-gradient signals, or components of advantage and temporal-difference estimates.

Read how Reinforcement Learning from Human Feedback explains action-value function →

What to know first

Related concepts

Questions about action-value function

What is action-value function?

An action-value function maps a state and action to the expected cumulative return from taking that action and then following a specified policy.

Which Manning books explain action-value function?

Applied Reinforcement Learning, Grokking AI Algorithms, Second Edition, Optimization Algorithms: AI techniques for design, planning, and control problems, Evolutionary Deep Learning: Genetic algorithms and neural networks, Build a Robo-Advisor with Python (From Scratch) and Reinforcement Learning from Human Feedback.

What should I know before learning action-value function?

Manning books that teach action-value function usually introduce dynamic programming, policy, reinforcement learning and exploitation first.

What is action-value function a kind of?

Action-value function is a kind of Bellman equation. A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.

What is action-value function part of?

Action-value function is part of Q-table. A Q-table is a lookup table storing estimated action values for state-action pairs in a tabular reinforcement-learning system.

What is action-value function also called?

Action-value function is also called q-function or q-value.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.