Radiotherapy treatment planning balances tumor control against radiation exposure to healthy organs. Treatment is typically divided into 20 to 40 sessions, or fractions. Dose accumulates, and the tumor may respond between sessions, making each beam and intensity decision relevant to later choices. section 8.4
A reinforcement learning simulation represents each decision using cumulative doses, remaining tumor dose, organ dose margins, tumor size, and treatment progress. An action selects a target and intensity. The reward favors progress toward the tumor dose target while penalizing organ-limit violations and tumor size. section 8.4
A simulated treatment
One environment models 30 fractions, five organs, and three intensity levels. It adds dose spill to neighboring organs and noise to delivered doses, then updates tumor size according to the dose received. A Double DQN agent learns decisions within this simulated setting; the example is not evidence of clinical effectiveness. section 8.4
Listing 8.5 Computing the reward signal
def _compute_reward(
self,
dose_change: np.ndarray,
done: bool,
success: bool,
failure: bool,
prev_tumor_dose: float,
):
tumor_dose = self.cum_doses[0]
oar_doses = self.cum_doses[1:]
oar_limits = self.organ_limits[1:]
oar_risk = self.organ_risk_weights[1:]
prev_error = np.abs(prev_tumor_dose - self.target_dose) #A
curr_error = np.abs(tumor_dose - self.target_dose) #A
tumor_error = curr_error / self.target_dose #B
tumor_reward = -1.5 * tumor_error #B
if curr_error < prev_error: #C
direction_bonus = 0.3 #C
else:
direction_bonus = -0.15 #C
violations = np.maximum(0.0, oar_doses - oar_limits) #D
weighted_violations = violations * oar_risk #E
organ_penalty = -0.3 * np.sum(weighted_violations) #E
tumor_size_penalty = -0.4 * self.tumor_size #F
step_penalty = -self.step_cost #G
reward = (
tumor_reward
+ direction_bonus
+ organ_penalty
+ tumor_size_penalty
+ step_penalty
) #H
if done:
if success:
reward += 200.0 #I
if failure:
reward -= 200.0 #J
return float(reward)
The reward calculation makes the trade-off between tumor dose, organ exposure, and tumor size explicit. section 8.4
