liveBook · Manning

Chapter 8: Deep Q-Networks for High-dimensional Data

What you will learn in chapter 8 of Applied Reinforcement Learning

This introduction uses radiotherapy planning to motivate moving beyond tabular reinforcement learning for complex, high-dimensional decisions.

After this chapter you will be able to

What it builds on

Section by section

Introduction

This introduction uses radiotherapy planning to motivate moving beyond tabular reinforcement learning for complex, high-dimensional decisions.

Key terms: state, action space, sequential decision-making, uncertainty, tabular methods, high-dimensional

8.1 From tabular to linear function approximation

This section explains why value-function approximation replaces tables in large state spaces and traces linear temporal-difference learning as a foundation for DQN.

Key terms: tabular methods, function approximation, representation error, generalization, feature extraction, feature vector, linear model, parameter vector, weighted sum, temporal-difference target

8.2 Deep learning fundamentals

This section explains how deep neural networks learn representations and predictions through forward passes, loss calculation, and backpropagation.

Key terms: forward pass, input layer, representations, output layer, Q-values, neuron, weighted sum, bias, nonlinear activation, ReLU

8.3 DQN theory: Q-learning meets neural networks

This section explains how DQN combines neural-network Q-learning with experience replay and a target network to stabilize learning.

Key terms: DQN, Q-learning, function approximation, Bellman equation, experience replay, replay buffer, transitions, temporal correlation, catastrophic forgetting, policy network

8.4 Implementation of DQN for treatment optimization

This section builds a simulated radiotherapy environment and introduces the DQN components used to learn sequential treatment decisions.

Key terms: state representation, cumulative doses, OAR safety margins, Q-values, Markovian, discrete actions, dose intensity, spillover, tumor dynamics, reward shaping

8.5 DQN beyond treatment: when and how to use DQN

This section explains where DQN applies beyond healthcare and how to recognize suitable problems and practical limitations.

Key terms: sequential decision-making, stochastic dynamics, long-term value, high-dimensional state spaces, representations, discrete action spaces, Q-network, continuous action spaces, sample inefficiency, reward hacking

8.6 Summary

Where this is used later

Concepts explained in this chapter

← Chapter 7: Monte Carlo Tree Search: Searching with Reinforcement Learning Principles Chapter 9: The Calculus of Decisions: Policy Gradient Methods →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.