Q-learning

As explained in Grokking AI Algorithms, Second Edition

Q-learning is a reinforcement-learning algorithm that learns action values for states without requiring a model of the environment's transition dynamics.

Q-learning is a way for an agent to learn which actions are promising in different situations. It is model-free: the agent does not need to know how the environment works in advance. Instead, it tries actions, receives rewards, observes what changes, and improves its estimates. section 10.6

Its Q-table works like a scorecard. Each row represents a state, each column an available action, and each cell a Q-value: an estimate of that action’s quality in that state. To use the table, the agent finds its current row and chooses the action with the highest score. section 10.6

Learning a route

Imagine an agent moving through a parking lot toward a goal. Its position is a state, and a move is an action. The table starts with guesses. As the agent tries moves and sees their rewards and resulting positions, it revises the scores. It sometimes tries a random move rather than following the current highest score, giving it a chance to learn from other actions. section 10.6

Figure 10.13 Life cycle of a Q-learning reinforcement learning algorithm

Follow the training cycle from choosing an action through observing its outcome and improving the Q-table. section 10.6

A simple implementation creates a table with one entry per state–action pair, runs repeated simulations, and chooses either the current highest-scoring action or a random one. Deep Q-learning changes where the estimates live: a neural network predicts Q-values instead of storing them in a table. section 10.6 section 10.10

How Grokking AI Algorithms, Second Edition covers it

How deep it goes

Explains the Q-table and action-selection cycle, then outlines repeated simulation in Python without developing a theoretical proof.

Examples

Uses parking-lot positions and moves, Q-table figures, update illustrations, and a Python simulation listing.

What it assumes

Defines states, actions, Q-values, and model-free learning in plain terms while using a Python listing for implementation.

What you can do after

Build it from scratch using a state–action table, simulated moves, rewards, and a choice between highest-scoring and random actions.

How it connects

Connects Q-learning to model-free reinforcement learning, reward feedback, Q-tables, and deep Q-learning.

liveBookmanning.com

Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksQ-learning is a model-free reinforcement-learning algorithm that learns action values by updating estimates of expected return for state-action pairs.
• 

  liveBookmanning.com
  Build a Robo-Advisor with Python (From Scratch)Q-learning is a model-free reinforcement-learning algorithm that estimates the value of taking each action in each state and uses those estimates to derive a policy.

Compare how 3 books explain Q-learning →

Related concepts

• reinforcement learningkind of · Reinforcement learning is a machine-learning paradigm in which an agent learns actions or a policy through interaction with an environment and evaluative feedback, such as rewards.
• epsilon-greedyuses · Epsilon-greedy is an action-selection strategy that chooses a random action with probability epsilon and otherwise selects the currently highest-valued action.
• deep reinforcement learningcontrasts with · Deep reinforcement learning is reinforcement learning that uses deep neural networks to represent policies, value functions, or other decision-making components.

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.