A radiotherapy plan involves repeated decisions: deliver enough radiation to control a tumor while limiting damage to healthy organs. Patient response is uncertain, and accumulated doses affect later choices. Storing a value for every possible state is impractical. A deep Q-network (DQN) replaces the Q-table with a neural network that estimates the value of actions and generalizes across states. section 8.0 section 8.3
DQN applies Q-learning rather than inventing a new learning rule. The network learns by adjusting its parameters to reduce errors in its value predictions. Naive training can be unstable: changing predictions also changes the targets used to train them, while consecutive experiences are highly correlated. section 8.3
A practical DQN learning loop uses several components:
- Select an action using ε-greedy exploration: sometimes choose randomly, and otherwise choose the action with the highest predicted value. Reduce the random-action rate during training. section 8.6 section 8.4
- Store transitions in an experience replay buffer and train on randomly sampled batches. Sampling helps break correlations between consecutive observations. section 8.6 section 8.4
- Use a separate target network to calculate learning targets, updating its parameters periodically. This makes targets less volatile than predictions from the continually updated online network. section 8.3 section 8.6
- In Double DQN, let the online network select the next action and the target network evaluate it. Separating these roles reduces overestimation bias. section 8.6
Figure 8.6 Deep Q-Networks learning systems
Follow the flow of experience, value estimates, and learning signals through the DQN system. section 8.3
Treatment optimization
A simulated treatment environment represents cumulative doses, remaining tumor dose, organ safety margins, tumor size, and treatment progress in its state. An action selects an organ and dose intensity. Rewards favor progress toward the tumor dose target and penalize organ-limit violations and tumor size; separate checks define simulated success and failure. The agent trains with replay, ε-greedy actions, a target network, and gradually increasing environment difficulty. section 8.4
Listing 8.13 Double DQN training step
def train_step(self):
if len(self.replay_buffer) < self.min_buffer_size: #A
return None #A
(
states,
actions,
rewards,
next_states,
dones,
) = self.replay_buffer.sample(self.batch_size) #B
q_values, cache = self.online_net.forward(states) #C
next_q_online = self.online_net.predict(next_states) #D
best_next_actions = np.argmax(next_q_online, axis=1) #E
next_q_target = self.target_net.predict(next_states) #F
best_next_q = next_q_target[
np.arange(self.batch_size), best_next_actions
] #G
targets = rewards + self.gamma * best_next_q * (~dones) #H
q_selected = q_values[np.arange(self.batch_size), ➥actions] #I
td_errors = q_selected - targets #I
grad_q = np.zeros_like(q_values) #J
grad_q[np.arange(self.batch_size), actions] = td_errors #K
self.online_net.update(states, grad_q, cache, self.lr) #L
if self.total_steps % self.target_update_freq == 0: #M
self.target_net.copy_from(self.online_net) #N
loss = 0.5 * np.mean(td_errors ** 2) #O
return loss
Inspect the Double DQN training step: the online network chooses the next action, and the target network evaluates it. section 8.4
The same decision pattern appears in inventory ordering, energy dispatch, pricing, and recommendations: actions affect future states, and immediate feedback does not capture the whole outcome. State representation, reward design, and safety checks remain important; a penalty for an undesirable outcome is not itself a guarantee that it cannot occur. section 8.5 section 8.4

