Q-learning is a model-free, off-policy method that estimates action values from experience. Its learning target assumes the best-valued action will be taken in the next state, even when the agent behaves exploratorily. section 6.4.2 section 6.7
A tabular Q-learning loop proceeds as follows:
- Initialize action values and choose an action, such as with an ε-greedy policy. section 6.4.2
- Take the action and observe the reward and next state. section 6.4.2
- Update the current action value using the maximum estimated value among actions in the next state; then continue from that state. section 6.4.2
Table 6.2. Sample Q-table for Restaurant Scheduling Problem
|
Hours Remaining (State)
|
Seat Immediately (Action 0)
|
Wait for Large Party (Action 1)
|
|
0 (Closed)
|
0.00
|
0.00
|
|
1 hour left
|
50.00
|
36.00
|
|
2 hours left
|
50.00
|
67.50
|
|
3 hours left
|
50.00
|
78.53
|
|
4 hours left
|
50.00
|
85.47
|
The restaurant-scheduling Q-table compares estimated action values by hours remaining, making the preferred action visible in each state. section 6.4
Restaurant scheduling
With two hours remaining, the table assigns a value of 67.50 to waiting for a large party and 50.00 to seating immediately. With one hour remaining, seating immediately has the higher value: 50.00 versus 36.00. section 6.4



