Q-learning
Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.
Evolutionary Deep Learning: Genetic algorithms and neural networks
Q-learning is a model-free reinforcement-learning algorithm that learns action values by updating estimates of expected return for state-action pairs.
- How it is used
- Uses action-value estimates to train an agent to navigate reward-based Gym environments.
- How deep it goes
- Walks through a runnable Q-table implementation, exploration, update rule, training loop, and evaluation before deferring complex environments to DQN.
- Examples
- Demonstrates a 4-by-4 Frozen Lake agent and relates deep Q-learning to CartPole, Atari, and Lunar Lander.
- What you can do after
- Build it from scratch as a Gym-based Q-table agent with epsilon exploration, reward updates, and greedy evaluation.
Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains Q-learning →
Applied Reinforcement Learning
Q-learning is a model-free reinforcement-learning algorithm that estimates the optimal action-value function through temporal-difference updates using the maximum estimated value of the next state.
- How it is used
- Learns action values toward an optimal policy while allowing exploratory behavior to generate experience.
- How deep it goes
- Works through initialization, action selection, observation, and a maximum-next-value update, without requiring the next action for learning.
- Examples
- Uses a restaurant-scheduling Q-table to compare waiting for a large party with seating immediately at different times.
- What you can do after
- Apply it by updating tabular action values from observed rewards and next states, using the highest estimated next-state value.
Read how Applied Reinforcement Learning explains Q-learning →
Grokking AI Algorithms, Second Edition
Q-learning is a reinforcement-learning algorithm that learns action values for states without requiring a model of the environment's transition dynamics.
- How it is used
- Introduces model-free reinforcement learning through an agent choosing actions in a simulated environment.
- How deep it goes
- Explains Q-table action selection and training, with example updates and a Python training routine.
- Examples
- Uses a moving agent in a state grid and a parking-lot problem to illustrate action values.
- What you can do after
- Practice training a simulated agent and selecting actions from learned Q-values.
Read how Grokking AI Algorithms, Second Edition explains Q-learning →
Build a Robo-Advisor with Python (From Scratch)
Q-learning is a model-free reinforcement-learning algorithm that estimates the value of taking each action in each state and uses those estimates to derive a policy.
- How it is used
- Deploys repeated simulated episodes to learn asset-allocation policies for goals-based financial planning.
- How deep it goes
- Explains the state-action update and implements exploration, transitions, initialization, and episode generation.
- Examples
- Uses wealth states, time horizons, asset-allocation actions, simulated transitions, terminal goals, and repeated episodes.
- What you can do after
- Apply it via Python to estimate Q values and derive policies from simulated financial episodes.
Read how Build a Robo-Advisor with Python (From Scratch) explains Q-learning →
Also covered in
What to know first
- dynamic programmingDynamic programming is an algorithmic technique that solves problems by decomposing them into overlapping subproblems and storing or reusing their solutions.
- policyA policy is a rule, principle, or decision function that governs behavior or determines actions or permissions under specified conditions.
- reinforcement learningReinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- actionAn action is a choice or operation selected or performed by an actor that can affect a system, environment, or process.
Related concepts
- off-policy methodkind of · An off-policy method learns or evaluates a target policy using experience generated by a different behavior policy.
- SARSA(λ)contrasts with · SARSA(λ) is an on-policy SARSA variant that uses eligibility traces to propagate temporal-difference credit to previously visited state-action pairs.
- reinforcement learningkind of · Reinforcement learning is a machine-learning paradigm in which an agent learns behavior or a decision policy through interaction with an environment and evaluative feedback, such as rewards.
- epsilon-greedyuses · Epsilon-greedy is an action-selection strategy that chooses a random action with a specified probability and otherwise chooses the action with the highest estimated value.
- deep reinforcement learningcontrasts with · Deep reinforcement learning is reinforcement learning that uses deep neural networks to represent policies, value functions, or other decision-making components.
- model-free methodkind of · A model-free method is a reinforcement-learning approach that learns policies or value functions directly from experience without an explicit model of environment dynamics and rewards.
Questions about Q-learning
What is Q-learning?
Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.
Which Manning books explain Q-learning?
Evolutionary Deep Learning: Genetic algorithms and neural networks, Applied Reinforcement Learning, Grokking AI Algorithms, Second Edition and Build a Robo-Advisor with Python (From Scratch).
What should I know before learning Q-learning?
Manning books that teach Q-learning usually introduce dynamic programming, policy, reinforcement learning and action first.
What is the difference between Q-learning and SARSA(λ)?
Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values. SARSA(λ) is an on-policy SARSA variant that uses eligibility traces to propagate temporal-difference credit to previously visited state-action pairs.
What is Q-learning a kind of?
Q-learning is a kind of off-policy method. An off-policy method learns or evaluates a target policy using experience generated by a different behavior policy.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



