liveBook · Manning

Q-learning

As explained in Applied Reinforcement Learning

Q-learning is a model-free reinforcement-learning algorithm that estimates the optimal action-value function through temporal-difference updates using the maximum estimated value of the next state.

Q-learning is a model-free, off-policy method that estimates action values from experience. Its learning target assumes the best-valued action will be taken in the next state, even when the agent behaves exploratorily. section 6.4.2 section 6.7

A tabular Q-learning loop proceeds as follows:

  • Initialize action values and choose an action, such as with an ε-greedy policy. section 6.4.2
  • Take the action and observe the reward and next state. section 6.4.2
  • Update the current action value using the maximum estimated value among actions in the next state; then continue from that state. section 6.4.2
Table 6.2. Sample Q-table for Restaurant Scheduling Problem
Hours Remaining (State)
Seat Immediately (Action 0)
Wait for Large Party (Action 1)
0 (Closed)
0.00
0.00
1 hour left
50.00
36.00
2 hours left
50.00
67.50
3 hours left
50.00
78.53
4 hours left
50.00
85.47

The restaurant-scheduling Q-table compares estimated action values by hours remaining, making the preferred action visible in each state. section 6.4

Restaurant scheduling

With two hours remaining, the table assigns a value of 67.50 to waiting for a large party and 50.00 to seating immediately. With one hour remaining, seating immediately has the higher value: 50.00 versus 36.00. section 6.4

How Applied Reinforcement Learning covers it

How deep it goes
Works through initialization, action selection, observation, and a maximum-next-value update, without requiring the next action for learning.
Examples
Uses a restaurant-scheduling Q-table to compare waiting for a large party with seating immediately at different times.
What it assumes
Builds on temporal-difference learning and action-value functions; no explicit prerequisite list is given.
What you can do after
Apply it by updating tabular action values from observed rewards and next states, using the highest estimated next-state value.
How it connects
Ties Q-learning to temporal-difference learning, SARSA, off-policy learning, and tabular reinforcement learning.

Also explained in

Compare how 4 books explain Q-learning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.