Action-value function
An action-value function maps a state and action to the expected cumulative return from taking that action and then following a specified policy.
Applied Reinforcement Learning
An action-value function maps a state-action pair to the expected return from taking the action and then following a policy.
- How it is used
- Uses action values to compare decisions during policy improvement and to select actions without a model.
- How deep it goes
- Develops Bellman one-step lookahead and greedy improvement, then explains direct tabular learning through temporal-difference updates.
- Examples
- Compares inventory order quantities and implements policy improvement for constrained project resource allocation.
- What you can do after
- Apply action-value comparisons to improve a policy and understand how tabular methods learn action values from experience.
Read how Applied Reinforcement Learning explains action-value function →
Grokking AI Algorithms, Second Edition
A Q-function maps a state-action pair to the expected cumulative reward obtained by taking the action and subsequently following a policy.
- How it is used
- Guides action selection toward cumulative rewards in Q-learning and deep Q-learning.
- How deep it goes
- Explains table-based scores and neural-network approximation, including a worked target and squared loss.
- Examples
- Uses parking-lot states and actions, plus an eastward move with four predicted next-state values.
- What you can do after
- Evaluate Q-values to choose a high-scoring action and calculate a neural-network training target from a transition.
Read how Grokking AI Algorithms, Second Edition explains action-value function →
Optimization Algorithms: AI techniques for design, planning, and control problems
An action-value function estimates the expected cumulative future reward from taking a specified action in a state and then following a policy.
- How it is used
- Frames action selection as choosing actions with the greatest expected cumulative discounted reward.
- How deep it goes
- Defines Q(s,a), contrasts it with V(s), and works through one expected-return calculation without developing the full learning algorithm.
- Examples
- Uses the Reach the Treasure game and a probability-weighted trajectory whose expected return equals 5.728.
- What you can do after
- Evaluate the long-term desirability of taking a particular action from a particular state under a policy.
Evolutionary Deep Learning: Genetic algorithms and neural networks
A Q-function maps a state and action to an estimate of the expected cumulative reward from taking that action and following a policy thereafter.
- How it is used
- Deploys Q-values to balance exploration and exploitation while learning action choices through repeated environment interactions.
- How deep it goes
- Explains table representation and update mechanics, but deliberately stops short of a deeper Q-learning treatment.
- Examples
- Uses a Gym FrozenLake environment, a 16-by-4 Q-table, training episodes, and greedy simulated episodes.
- What you can do after
- Build it from scratch as a tabular Q-learning agent and evaluate its learned actions in an environment.
Build a Robo-Advisor with Python (From Scratch)
A Q-function maps a state and an action to the estimated cumulative reward obtained by taking that action and following a policy afterward.
- How it is used
- Deploys state-action estimates to learn asset-allocation policies for financial planning.
- How deep it goes
- Explains the update equation and implementation loop, stopping short of broader algorithmic variations.
- Examples
- Uses wealth-time-action arrays, random wealth transitions, terminal goals, exploratory action selection, and repeated Python episodes.
- What you can do after
- Builds a Q-learning loop that estimates action values and supports policy selection through exploration and exploitation.
Read how Build a Robo-Advisor with Python (From Scratch) explains action-value function →
Reinforcement Learning from Human Feedback
A Q-function maps a state-action pair to a scalar estimate of expected future reward.
- How it is used
- Uses state–action reward estimates to support Q-learning, temporal-difference updates, and policy-gradient learning signals.
- How deep it goes
- Defines Q, derives its Bellman relation to V, and connects it to advantages and TD residuals without developing a full Q-learning algorithm.
- Examples
- Uses deterministic language-model transitions, game-playing systems, adaptive control, and a policy-gradient TD-residual calculation.
- What you can do after
- Apply Q-values as Bellman targets, policy-gradient signals, or components of advantage and temporal-difference estimates.
Read how Reinforcement Learning from Human Feedback explains action-value function →
What to know first
- dynamic programmingDynamic programming is an algorithmic technique that solves problems by decomposing them into overlapping subproblems and storing or reusing their solutions.
- policyA policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- exploitationIn optimization and decision-making, exploitation is selecting an option currently believed, based on available information, to yield high value.
Related concepts
- SARSA(λ)used by · SARSA(λ) is an on-policy SARSA variant that uses eligibility traces to propagate temporal-difference credit to previously visited state-action pairs.
- Bellman equationkind of · A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
- Q-learningused by · Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.
- Q-tablepart of · A Q-table is a lookup table storing estimated action values for state-action pairs in a tabular reinforcement-learning system.
- value functionkind of · A value function estimates the expected cumulative reward, utility, or quality associated with a state, action, or policy.
- advantage functionused by · An advantage function measures the expected return of an action relative to a baseline, typically the expected return under a policy at the same state.
Questions about action-value function
What is action-value function?
An action-value function maps a state and action to the expected cumulative return from taking that action and then following a specified policy.
Which Manning books explain action-value function?
Applied Reinforcement Learning, Grokking AI Algorithms, Second Edition, Optimization Algorithms: AI techniques for design, planning, and control problems, Evolutionary Deep Learning: Genetic algorithms and neural networks, Build a Robo-Advisor with Python (From Scratch) and Reinforcement Learning from Human Feedback.
What should I know before learning action-value function?
Manning books that teach action-value function usually introduce dynamic programming, policy, reinforcement learning and exploitation first.
What is action-value function a kind of?
Action-value function is a kind of Bellman equation. A Bellman equation recursively relates a state or action value to an immediate reward and the expected discounted value of successor states or actions.
What is action-value function part of?
Action-value function is part of Q-table. A Q-table is a lookup table storing estimated action values for state-action pairs in a tabular reinforcement-learning system.
What is action-value function also called?
Action-value function is also called q-function or q-value.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.





