liveBook · Manning

Q-learning

Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.

How 4 Manning books explain it

Evolutionary Deep Learning: Genetic algorithms and neural networks

Q-learning is a model-free reinforcement-learning algorithm that learns action values by updating estimates of expected return for state-action pairs.

How it is used
Uses action-value estimates to train an agent to navigate reward-based Gym environments.
How deep it goes
Walks through a runnable Q-table implementation, exploration, update rule, training loop, and evaluation before deferring complex environments to DQN.
Examples
Demonstrates a 4-by-4 Frozen Lake agent and relates deep Q-learning to CartPole, Atari, and Lunar Lander.
What you can do after
Build it from scratch as a Gym-based Q-table agent with epsilon exploration, reward updates, and greedy evaluation.

Read how Evolutionary Deep Learning: Genetic algorithms and neural networks explains Q-learning →

Applied Reinforcement Learning

Q-learning is a model-free reinforcement-learning algorithm that estimates the optimal action-value function through temporal-difference updates using the maximum estimated value of the next state.

How it is used
Learns action values toward an optimal policy while allowing exploratory behavior to generate experience.
How deep it goes
Works through initialization, action selection, observation, and a maximum-next-value update, without requiring the next action for learning.
Examples
Uses a restaurant-scheduling Q-table to compare waiting for a large party with seating immediately at different times.
What you can do after
Apply it by updating tabular action values from observed rewards and next states, using the highest estimated next-state value.

Read how Applied Reinforcement Learning explains Q-learning →

Grokking AI Algorithms, Second Edition

Q-learning is a reinforcement-learning algorithm that learns action values for states without requiring a model of the environment's transition dynamics.

How it is used
Introduces model-free reinforcement learning through an agent choosing actions in a simulated environment.
How deep it goes
Explains Q-table action selection and training, with example updates and a Python training routine.
Examples
Uses a moving agent in a state grid and a parking-lot problem to illustrate action values.
What you can do after
Practice training a simulated agent and selecting actions from learned Q-values.

Read how Grokking AI Algorithms, Second Edition explains Q-learning →

Build a Robo-Advisor with Python (From Scratch)

Q-learning is a model-free reinforcement-learning algorithm that estimates the value of taking each action in each state and uses those estimates to derive a policy.

How it is used
Deploys repeated simulated episodes to learn asset-allocation policies for goals-based financial planning.
How deep it goes
Explains the state-action update and implements exploration, transitions, initialization, and episode generation.
Examples
Uses wealth states, time horizons, asset-allocation actions, simulated transitions, terminal goals, and repeated episodes.
What you can do after
Apply it via Python to estimate Q values and derive policies from simulated financial episodes.

Read how Build a Robo-Advisor with Python (From Scratch) explains Q-learning →

Also covered in

What to know first

Related concepts

Questions about Q-learning

What is Q-learning?

Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values.

Which Manning books explain Q-learning?

Evolutionary Deep Learning: Genetic algorithms and neural networks, Applied Reinforcement Learning, Grokking AI Algorithms, Second Edition and Build a Robo-Advisor with Python (From Scratch).

What should I know before learning Q-learning?

Manning books that teach Q-learning usually introduce dynamic programming, policy, reinforcement learning and action first.

What is the difference between Q-learning and SARSA(λ)?

Q-learning is a model-free reinforcement-learning algorithm that estimates state-action values through updates based on rewards and subsequent state values. SARSA(λ) is an on-policy SARSA variant that uses eligibility traces to propagate temporal-difference credit to previously visited state-action pairs.

What is Q-learning a kind of?

Q-learning is a kind of off-policy method. An off-policy method learns or evaluates a target policy using experience generated by a different behavior policy.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.