Curriculum learning trains an agent first in a simplified environment, then gradually increases difficulty. It gives the agent a chance to learn useful behavior before facing the full challenge. section 8.6
A curriculum can advance on a fixed schedule or in response to performance. One performance-based guideline is to increase difficulty when the success rate exceeds 70% for 100 or more episodes. Possible difficulty controls include noise, obstacles, and episode length. section 8.4 section 8.6
Radiotherapy treatment simulation
A DQN agent learns to choose radiation doses while approaching a tumor dose target and limiting exposure to organs at risk. The simulation starts at difficulty 0 and increases it linearly to 1 over 1,000 training episodes. As difficulty rises, dose noise increases, the overdose failure threshold tightens, and the organ-dose margin required for success becomes stricter. Tumor growth and shrinkage rates also change. section 8.4
Listing 8.7 Adjusting environment difficulty
def set_difficulty(self, difficulty: float):
difficulty = float(np.clip(difficulty, 0.0, 1.0)) #A
self.dose_noise_std = 0.05 + 0.25 * difficulty #B
self.tumor_growth_rate = 0.0 + 0.01 * difficulty #C
self.tumor_shrink_rate = 0.015 + 0.02 * difficulty #C
self.max_overdose_factor = 1.5 - 0.3 * difficulty #D
self.success_margin_factor = 1.25 - 0.25 * difficulty #D
The difficulty setter specifies how noise, tumor dynamics, and dose thresholds change with difficulty. section 8.4
Listing 8.14 Main training loop with curriculum
def main():
env = TreatmentEnv() #A
agent = DQNAgent(
state_dim=env.state_dim,
action_dim=env.action_dim,
hidden_dim=128,
gamma=0.99,
lr=1e-3,
buffer_capacity=50000,
batch_size=64,
min_buffer_size=1000,
target_update_freq=1000,
epsilon_start=1.0,
epsilon_end=0.03,
epsilon_decay_frames=50000,
seed=42,
) #B
num_episodes = 10000 #C
curriculum_episodes = 1000 #D
rewards_history = [], success_history = [], losses_history = []
for episode in range(num_episodes):
difficulty = min(1.0, episode
➥/ float(curriculum_episodes)) #E
env.set_difficulty(difficulty) #F
state = env.reset() #G
episode_reward = 0.0
success_flag = False
for t in range(env.n_fractions): #H
action = agent.select_action(state, greedy=False) #I
next_state, reward, done, info = env.step(action) #J
agent.push_transition(state, action, reward,
➥next_state, done) #K
loss = agent.train_step() #L
if loss is not None:
losses_history.append(loss)
episode_reward += reward
state = next_state
if done:
success_flag = info["success"]
break
rewards_history.append(episode_reward)
success_history.append(1.0 if success_flag else 0.0)
if (episode + 1) % 20 == 0: #M
avg_reward = np.mean(rewards_history[-20:])
avg_success = np.mean(success_history[-20:])
print(
f"Episode {episode+1:4d} | "
f"AvgReward(20): {avg_reward:8.2f} | "f"Success(20):
➥{avg_success*100:5.1f}% |
➥"f"Epsilon: {agent.epsilon:5.3f} | "
➥f"Difficulty: {difficulty:4.2f}") #N
The training loop updates difficulty before resetting the environment for each episode. section 8.4

