liveBook · Manning

DQN

As explained in Applied Reinforcement Learning

Deep Q-network (DQN) is a reinforcement-learning algorithm that uses a neural network to approximate action values for Q-learning.

A radiotherapy plan involves repeated decisions: deliver enough radiation to control a tumor while limiting damage to healthy organs. Patient response is uncertain, and accumulated doses affect later choices. Storing a value for every possible state is impractical. A deep Q-network (DQN) replaces the Q-table with a neural network that estimates the value of actions and generalizes across states. section 8.0 section 8.3

DQN applies Q-learning rather than inventing a new learning rule. The network learns by adjusting its parameters to reduce errors in its value predictions. Naive training can be unstable: changing predictions also changes the targets used to train them, while consecutive experiences are highly correlated. section 8.3

A practical DQN learning loop uses several components:

  • Select an action using ε-greedy exploration: sometimes choose randomly, and otherwise choose the action with the highest predicted value. Reduce the random-action rate during training. section 8.6 section 8.4
  • Store transitions in an experience replay buffer and train on randomly sampled batches. Sampling helps break correlations between consecutive observations. section 8.6 section 8.4
  • Use a separate target network to calculate learning targets, updating its parameters periodically. This makes targets less volatile than predictions from the continually updated online network. section 8.3 section 8.6
  • In Double DQN, let the online network select the next action and the target network evaluate it. Separating these roles reduces overestimation bias. section 8.6
Figure 8.6 Deep Q-Networks learning systems

Follow the flow of experience, value estimates, and learning signals through the DQN system. section 8.3

Treatment optimization

A simulated treatment environment represents cumulative doses, remaining tumor dose, organ safety margins, tumor size, and treatment progress in its state. An action selects an organ and dose intensity. Rewards favor progress toward the tumor dose target and penalize organ-limit violations and tumor size; separate checks define simulated success and failure. The agent trains with replay, ε-greedy actions, a target network, and gradually increasing environment difficulty. section 8.4

Listing 8.13 Double DQN training step
def train_step(self):
    if len(self.replay_buffer) < self.min_buffer_size:  #A
        return None  #A
 
    (
        states,
        actions,
        rewards,
        next_states,
        dones,
    ) = self.replay_buffer.sample(self.batch_size)  #B
 
    q_values, cache = self.online_net.forward(states)  #C
 
    next_q_online = self.online_net.predict(next_states)  #D
    best_next_actions = np.argmax(next_q_online, axis=1)  #E
 
    next_q_target = self.target_net.predict(next_states)  #F
    best_next_q = next_q_target[
        np.arange(self.batch_size), best_next_actions
    ]  #G
 
    targets = rewards + self.gamma * best_next_q * (~dones)  #H
 
    q_selected = q_values[np.arange(self.batch_size), ➥actions]  #I
    td_errors = q_selected - targets  #I
 
    grad_q = np.zeros_like(q_values)  #J
    grad_q[np.arange(self.batch_size), actions] = td_errors  #K
 
    self.online_net.update(states, grad_q, cache, self.lr)  #L
 
    if self.total_steps % self.target_update_freq == 0:  #M
        self.target_net.copy_from(self.online_net)  #N
 
    loss = 0.5 * np.mean(td_errors ** 2)  #O
    return loss

Inspect the Double DQN training step: the online network chooses the next action, and the target network evaluates it. section 8.4

The same decision pattern appears in inventory ordering, energy dispatch, pricing, and recommendations: actions affect future states, and immediate feedback does not capture the whole outcome. State representation, reward design, and safety checks remain important; a penalty for an undesirable outcome is not itself a guarantee that it cannot occur. section 8.5 section 8.4

How Applied Reinforcement Learning covers it

How deep it goes
Develops the instability rationale, stabilization mechanisms, and a worked implementation with network updates and a training loop.
Examples
Works through simulated radiotherapy planning and identifies inventory ordering, energy dispatch, dynamic pricing, trading, and recommendation applications.
What it assumes
Builds on Q-learning, function approximation, and neural-network foundations introduced earlier.
What you can do after
Builds a DQN agent from scratch for a simulated treatment environment, including replay, action selection, network updates, and a curriculum.
How it connects
Connects Q-learning and neural function approximation with experience replay, target networks, epsilon-greedy exploration, and Double DQN.

Also explained in

Compare how 2 books explain DQN →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.