Chapter 8: Deep Q-Networks for High-dimensional Data
This introduction uses radiotherapy planning to motivate moving beyond tabular reinforcement learning for complex, high-dimensional decisions.
After this chapter you will be able to
- Limits of tabular values. Explain why storing a separate value for every possible patient state is impractical in the radiotherapy example.
- Tabular and approximation tradeoffs. Explain why large or continuous state spaces make tabular values impractical and what function approximation trades for tractability.
- Linear feature value estimates. Describe how features and weights produce a linear value estimate, and distinguish evaluating the model from updating it.
- Bootstrapped target and error. Trace how a reward and next state produce a temporal-difference target and error for a linear action-value estimate.
- Feature weighted parameter updates. Explain how TD error, features, and the learning rate update the linear model and allow one experience to affect similar situations.
- Forward Pass and Representations. Trace how a deep network transforms raw input into a prediction and learns representations across its layers.
- Layer Transformations and Nonlinearity. Explain how weighted sums and activation functions work together in network layers, and why nonlinearity matters.
- Loss and Learning Targets. Explain how a loss compares predictions with feedback, distinguishing supervised labels from reinforcement learning targets.
- Backpropagation and Parameter Updates. Explain how backpropagation and an optimizer use the loss to adjust a deep network's parameters.
- Neural Q-value approximation. Explain how DQN replaces tabular Q-values with a neural network and why naive neural Q-learning can be unstable.
- Experience replay buffer. Describe how transitions are stored and sampled for experience replay, and explain how this helps learning.
- Target network stabilization. Explain the separate roles of the policy and target networks and how periodic copying stabilizes targets.
- Bellman target and regression. Trace how a sampled transition produces a DQN target and how prediction error updates the policy network.
- Exploration and off-policy learning. Explain the roles of ε-greedy action selection and off-policy learning in DQN.
- Treatment state and objectives. Explain how the treatment problem is represented as a sequential decision process and how its state supports the Q-network.
- Action and treatment dynamics. Trace how a discrete action affects dose and tumor response, and explain why those effects matter for later fractions.
- Treatment reward shaping. Explain how the reward combines treatment objectives and why step-by-step shaping helps learning.
- Treatment difficulty curriculum. Explain how the training curriculum changes the treatment task and why starting easier helps.
- Replay buffer training data. Describe what the replay buffer stores and how its sampling supports DQN training.
- Sequential decision making applications. Explain what makes sequential problems a good fit for DQN, using an application beyond healthcare as an example.
- State and action suitability. Explain how state and action space characteristics influence whether DQN is a suitable choice.
- Interaction and reward limitations. Explain how costly interactions and poorly specified rewards can limit DQN in practice.
What it builds on
- Markov decision process (chapter 2) · Sequential decisions can be framed in terms of states and actions.
- Tabular Reinforcement Learning: Q-learning and SARSA (chapter 6) · Tabular reinforcement learning represents values in tables.
- Q-learning: learning from what you should do (chapter 6) · Q-learning estimates action values for state–action pairs.
- Temporal difference learning (chapter 6) · Temporal-difference learning uses experience and current predictions to learn values.
- Tradeoff between exploitation and exploration (chapter 5) · Exploration and exploitation have different roles in decision making.
- Off policy vs on policy learning (chapter 6) · Off-policy learning separates the behavior policy from the policy being learned.
- State: Anatomy of sequential decision making (chapter 2) · States summarize the context needed for sequential decisions.
- Design rewards to be stepwise, whenever possible (chapter 2) · Stepwise rewards provide feedback during an episode.
Section by section
Introduction
This introduction uses radiotherapy planning to motivate moving beyond tabular reinforcement learning for complex, high-dimensional decisions.
- Sequential radiotherapy decision problem: Describe radiotherapy planning as a sequential decision problem under uncertainty, including the treatment tradeoff and how patient response changes over time.
Why it matters: This example shows why treatment planning involves a sequence of uncertain, interrelated decisions. - Limits of tabular values: Explain why storing a separate value for every possible patient state is impractical in the radiotherapy example.
Why it matters: Recognizing the scale of the state space explains why a table of values is not a practical solution here.
Key terms: state, action space, sequential decision-making, uncertainty, tabular methods, high-dimensional
8.1 From tabular to linear function approximation
This section explains why value-function approximation replaces tables in large state spaces and traces linear temporal-difference learning as a foundation for DQN.
- Tabular and approximation tradeoffs: Explain why large or continuous state spaces make tabular values impractical and what function approximation trades for tractability.
Why it matters: This tradeoff explains why an agent may need a model rather than a table to handle realistic states. - Linear feature value estimates: Describe how features and weights produce a linear value estimate, and distinguish evaluating the model from updating it.
Why it matters: This separates the value prediction from the later learning step that changes the model. - Bootstrapped target and error: Trace how a reward and next state produce a temporal-difference target and error for a linear action-value estimate.
Why it matters: The TD error turns one observed transition into a signal for improving the value model. - Feature weighted parameter updates: Explain how TD error, features, and the learning rate update the linear model and allow one experience to affect similar situations.
Why it matters: This update shows how feedback reshapes a shared value function rather than just one table entry. - Linear limits and DQN: Explain why linear value models depend on feature quality and what DQN changes at a high level.
Why it matters: This identifies the representation problem that motivates moving beyond linear models.
Key terms: tabular methods, function approximation, representation error, generalization, feature extraction, feature vector, linear model, parameter vector, weighted sum, temporal-difference target
8.2 Deep learning fundamentals
This section explains how deep neural networks learn representations and predictions through forward passes, loss calculation, and backpropagation.
- Forward Pass and Representations: Trace how a deep network transforms raw input into a prediction and learns representations across its layers.
Why it matters: This explains how a network turns a complex state into useful action-value predictions without hand-designing every feature. - Layer Transformations and Nonlinearity: Explain how weighted sums and activation functions work together in network layers, and why nonlinearity matters.
Why it matters: Nonlinear activations let a deep network represent relationships that stacked linear transformations alone cannot. - Hidden and Output Activations: Describe the activation choices commonly used for hidden layers and Q-value outputs in deep reinforcement learning.
Why it matters: These choices connect nonlinear hidden processing with outputs that can represent value estimates without a fixed bound. - Loss and Learning Targets: Explain how a loss compares predictions with feedback, distinguishing supervised labels from reinforcement learning targets.
Why it matters: The loss provides the feedback signal that makes it possible to improve predictions. - Backpropagation and Parameter Updates: Explain how backpropagation and an optimizer use the loss to adjust a deep network's parameters.
Why it matters: Backpropagation makes end-to-end learning practical across many layers and parameters.
Key terms: forward pass, input layer, representations, output layer, Q-values, neuron, weighted sum, bias, nonlinear activation, ReLU
8.3 DQN theory: Q-learning meets neural networks
This section explains how DQN combines neural-network Q-learning with experience replay and a target network to stabilize learning.
- Neural Q-value approximation: Explain how DQN replaces tabular Q-values with a neural network and why naive neural Q-learning can be unstable.
Why it matters: This explains both why DQN scales beyond a table and why it needs stabilization. - Experience replay buffer: Describe how transitions are stored and sampled for experience replay, and explain how this helps learning.
Why it matters: Replay makes updates less dependent on the latest run of similar experiences. - Target network stabilization: Explain the separate roles of the policy and target networks and how periodic copying stabilizes targets.
Why it matters: A delayed target gives the optimizer a more stable objective to pursue. - Bellman target and regression: Trace how a sampled transition produces a DQN target and how prediction error updates the policy network.
Why it matters: This connects the stored experience and the two networks to the actual learning update. - Exploration and off-policy learning: Explain the roles of ε-greedy action selection and off-policy learning in DQN.
Why it matters: This explains why DQN can explore while still learning from a changing history of behavior.
Key terms: DQN, Q-learning, function approximation, Bellman equation, experience replay, replay buffer, transitions, temporal correlation, catastrophic forgetting, policy network
8.4 Implementation of DQN for treatment optimization
This section builds a simulated radiotherapy environment and introduces the DQN components used to learn sequential treatment decisions.
- Treatment state and objectives: Explain how the treatment problem is represented as a sequential decision process and how its state supports the Q-network.
Why it matters: The state gives the agent the treatment context it needs to balance tumor control and organ safety over time. - Action and treatment dynamics: Trace how a discrete action affects dose and tumor response, and explain why those effects matter for later fractions.
Why it matters: A fraction affects both immediate treatment progress and the safety of future choices. - Treatment reward shaping: Explain how the reward combines treatment objectives and why step-by-step shaping helps learning.
Why it matters: The reward translates competing clinical objectives into feedback the agent can learn from. - Episode success and failure: Describe the environment’s success and failure checks and what happens when treatment runs out of fractions.
Why it matters: The termination rules distinguish successful treatment from severe toxicity and an unfinished plan. - Treatment difficulty curriculum: Explain how the training curriculum changes the treatment task and why starting easier helps.
Why it matters: An easier starting task helps the agent encounter useful outcomes before learning the full problem. - Replay buffer training data: Describe what the replay buffer stores and how its sampling supports DQN training.
Why it matters: Replay gives the Q-network varied past experience rather than a tightly correlated sequence of recent steps.
Key terms: state representation, cumulative doses, OAR safety margins, Q-values, Markovian, discrete actions, dose intensity, spillover, tumor dynamics, reward shaping
8.5 DQN beyond treatment: when and how to use DQN
This section explains where DQN applies beyond healthcare and how to recognize suitable problems and practical limitations.
- Sequential decision making applications: Explain what makes sequential problems a good fit for DQN, using an application beyond healthcare as an example.
Why it matters: These shared features help you recognize DQN opportunities in different domains. - State and action suitability: Explain how state and action space characteristics influence whether DQN is a suitable choice.
Why it matters: State and action space shape can make DQN practical or rule it out. - Interaction and reward limitations: Explain how costly interactions and poorly specified rewards can limit DQN in practice.
Why it matters: A suitable state and action space alone does not guarantee that DQN can be trained affordably or toward the right objective.
Key terms: sequential decision-making, stochastic dynamics, long-term value, high-dimensional state spaces, representations, discrete action spaces, Q-network, continuous action spaces, sample inefficiency, reward hacking
8.6 Summary
Where this is used later
- (chapter intro) (chapter 9) · DQN uses a neural network to represent Q-values.
- Why optimizing the policy directly? (chapter 9) · DQN estimates action values using a neural network.
- The reward is a specification the agent will exploit (chapter 12) · Connects reward design to the ways an agent may exploit its specification.
Concepts explained in this chapter
- DQNDeep Q-network (DQN) is a reinforcement-learning algorithm that uses a neural network to approximate action values for Q-learning.
- experience replayExperience replay is a reinforcement-learning technique that stores interaction transitions and trains on randomly sampled mini-batches to reduce temporal correlation.
- replay bufferA replay buffer is a storage structure that retains interaction records for later processing or learning updates.
- curriculum learningCurriculum learning trains a model by beginning with simpler tasks or environments and progressively increasing their difficulty.
- feedforward networkA feedforward neural network is a neural network in which information flows from input to output without recurrent connections.
- organ at riskAn organ at risk is a healthy organ near a radiation target whose exposure must be kept below a clinical dose limit.
- radiation doseThe amount of ionizing radiation delivered to a patient or specified tissue during treatment.
- radiation toxicityHarmful tissue or organ injury caused by radiation exposure.
- radiotherapyRadiotherapy is a cancer treatment that uses ionizing radiation to damage and destroy tumor cells.
- radiotherapy treatment planningRadiotherapy treatment planning is the process of selecting radiation delivery parameters over treatment sessions to control a tumor while limiting exposure to healthy tissue.
- tumorA tumor is an abnormal mass of tissue formed by uncontrolled cell growth and targeted for destruction in radiotherapy.
- function approximatorFunction approximation estimates a value function or policy with a parameterized mathematical function instead of storing a separate value for every state or state-action pair.
- deep neural networkA deep neural network is a neural network with multiple learned layers between its input and output.
- target networkA target network is a delayed copy of an action-value network whose parameters are updated periodically and used to compute learning targets.
- TD-learningTemporal-difference learning updates value estimates incrementally using observed rewards and estimates of subsequent values.
- feature extractionFeature extraction is the transformation of raw state information into numerical features that represent properties relevant to prediction.
- feature vectorA feature vector is an ordered collection of numerical features representing an input state or state-action pair.
- learning rateA learning rate is a numerical step-size parameter that controls how strongly an update changes an estimate or model parameter.
- linear function approximationA function-approximation technique that estimates values as weighted sums of state or state-action features.
- tabular methodTabular reinforcement learning stores value estimates or policy information explicitly in lookup tables indexed by states or state-action pairs.
- temporal-difference errorA temporal-difference error is the difference between a target formed from an immediate reward and a discounted estimate of future value and the current value estimate.
- backpropagationBackpropagation computes gradients of a model loss with respect to parameters by applying the chain rule backward through a computational graph.
- online networkThe currently updated neural network in DQN that estimates action values and supports action selection.
- reward hackingReward hacking is the exploitation of a specified reward function in ways that increase its numerical value without achieving the intended objective.
- activation functionA function that transforms a neural network unit's pre-activation value, typically introducing nonlinearity.
- chain ruleThe chain rule is a calculus rule for differentiating a composition of functions by multiplying derivatives of its successive components.
- deep learningDeep learning is a machine-learning approach that learns hierarchical representations and predictions using neural networks with multiple learned layers.
- forward passA forward pass computes a neural network's output by propagating an input through its layers in order.
- gradientA gradient is a vector of partial derivatives that indicates how a scalar function changes with respect to multiple variables.
- linear transformationA linear transformation maps inputs to weighted combinations of those inputs, possibly with an added bias.
- neural network layerA neural network layer is a computational stage that transforms its inputs into outputs using parameters and an activation operation.
- nonlinear mappingNonlinearity is a mathematical property that allows a model's output to vary in ways that cannot be represented by a single linear transformation.
- objective functionA mathematical quantity that a learning or optimization procedure seeks to minimize or maximize.
- optimizerAn optimizer is an algorithm that updates model parameters using an objective function and, commonly, its gradients.
- neural networkA neural network is a parameterized computational model composed of interconnected transformations that map inputs to outputs.
- action spaceAn action space is the set or structured specification of actions that an agent can submit to an environment.
- deep reinforcement learningA reinforcement-learning approach that uses deep neural networks to approximate policies, value functions, or related decision components.
- linear modelA linear model represents an outcome as a weighted linear combination of input features.
- bootstrappingBootstrapping is an estimation technique that updates a prediction using another prediction or value estimate rather than waiting for a complete realized outcome.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
