liveBook · Manning

Chapter 6: Tabular Reinforcement Learning

What you will learn in chapter 6 of Applied Reinforcement Learning

The introduction positions temporal-difference learning as a foundation of reinforcement learning and briefly contrasts tabular and approximation-based methods.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction positions temporal-difference learning as a foundation of reinforcement learning and briefly contrasts tabular and approximation-based methods.

Key terms: generalized policy iteration, policy evaluation, policy improvement, temporal-difference learning, tabular methods, function approximation, deep reinforcement learning, V(s), Q(s, a)

6.1 Temporal difference learning

This section introduces temporal-difference learning as an incremental, experience-based way to update value estimates without knowing the full future or environment probabilities.

Key terms: prediction error, learning rate, loss function, immediate reward, labels, sequential decision-making, temporal-difference learning, TD error, discounted value, value of the current state

6.2 A concrete example: restaurant table scheduling

This section uses restaurant table scheduling to compare temporal-difference learning from experience with an analytically calculated optimal solution.

Key terms: backward induction, expected value, discount factor, state, temporal-difference learning, convergence, episodes, value estimates, optimal value function, temporal-difference error

6.3 Off policy vs on policy learning

This section explains why model-free TD learning uses action values and distinguishes learning under the behavior policy from learning about a different policy.

Key terms: state values, action values, V(s), Q(s,a), model-free, experience tuples, sampling, state, action, reward

6.4 Tabular Reinforcement Learning: Q-learning and SARSA

This section introduces tabular action-value representation and contrasts the next-action choices used by SARSA and Q-learning.

Key terms: SARSA, Q-learning, temporal-difference learning, TD error, action-value function, tabular, lookup table, Q-table, state-action pair, Q-value

6.4.1 SARSA: learning from what you actually do

This section explains SARSA’s on-policy learning process, action selection, and practical tradeoffs.

Key terms: SARSA, State-Action-Reward-State-Action, on-policy, Q(s,a), Q(s',a'), terminal state, episode, next action, ε-greedy, exploitation

6.4.2 Q-learning: learning from what you should do

This section explains Q-learning’s off-policy update, its basic workflow, and its advantages and risks.

Key terms: Q-learning, off-policy learning, behavior policy, greedy policy, Q(s,a), ε-greedy, optimistic bootstrapping, maximum Q-value, batch learning, overoptimism

6.5 TD(λ) and Eligibility traces

This section explains how TD(λ) uses eligibility traces to distribute credit over earlier decisions and extends the approach to SARSA and Q-learning.

Key terms: TD(0), Monte Carlo, TD(λ), n-step returns, bias–variance tradeoff, eligibility trace, credit assignment, TD error, forward view, backward view

6.6 Gas station fuel purchase scheduling with tabular methods

This section models gas station fuel purchasing as an episodic inventory problem, implements tabular agents, and compares their business performance.

Key terms: current_inventory, pending_sum, next_arrival_time, Markovian, state space, action space, reward function, stockout penalties, holding costs, ordering costs

6.7 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 5: Contextual Bandit: Optimizing Stochastic One-step Decisions Chapter 7: Monte Carlo Tree Search: Searching with Reinforcement Learning Principles →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.