liveBook · Manning

Curriculum learning

As explained in Applied Reinforcement Learning

Curriculum learning trains a model by beginning with simpler tasks or environments and progressively increasing their difficulty.

Curriculum learning trains an agent first in a simplified environment, then gradually increases difficulty. It gives the agent a chance to learn useful behavior before facing the full challenge. section 8.6

A curriculum can advance on a fixed schedule or in response to performance. One performance-based guideline is to increase difficulty when the success rate exceeds 70% for 100 or more episodes. Possible difficulty controls include noise, obstacles, and episode length. section 8.4 section 8.6

Radiotherapy treatment simulation

A DQN agent learns to choose radiation doses while approaching a tumor dose target and limiting exposure to organs at risk. The simulation starts at difficulty 0 and increases it linearly to 1 over 1,000 training episodes. As difficulty rises, dose noise increases, the overdose failure threshold tightens, and the organ-dose margin required for success becomes stricter. Tumor growth and shrinkage rates also change. section 8.4

Listing 8.7 Adjusting environment difficulty
   def set_difficulty(self, difficulty: float):
        difficulty = float(np.clip(difficulty, 0.0, 1.0)) #A
        
        self.dose_noise_std = 0.05 + 0.25 * difficulty #B
 
        self.tumor_growth_rate = 0.0 + 0.01 * difficulty #C
        self.tumor_shrink_rate = 0.015 + 0.02 * difficulty #C
 
        self.max_overdose_factor = 1.5 - 0.3 * difficulty #D
        self.success_margin_factor = 1.25 - 0.25 * difficulty #D

The difficulty setter specifies how noise, tumor dynamics, and dose thresholds change with difficulty. section 8.4

Listing 8.14 Main training loop with curriculum
def main():
    env = TreatmentEnv()  #A
    agent = DQNAgent(
        state_dim=env.state_dim,
        action_dim=env.action_dim,
        hidden_dim=128,
        gamma=0.99,
        lr=1e-3,
        buffer_capacity=50000,
        batch_size=64,
        min_buffer_size=1000,
        target_update_freq=1000,
        epsilon_start=1.0,
        epsilon_end=0.03,
        epsilon_decay_frames=50000,
        seed=42,
    )  #B
 
    num_episodes = 10000  #C
    curriculum_episodes = 1000  #D
 
    rewards_history = [], success_history = [], losses_history = []
 
    for episode in range(num_episodes):
        difficulty = min(1.0, episode
        ➥/ float(curriculum_episodes))  #E
        env.set_difficulty(difficulty)  #F
 
        state = env.reset()  #G
        episode_reward = 0.0
        success_flag = False
 
        for t in range(env.n_fractions):  #H
            action = agent.select_action(state, greedy=False)  #I
            next_state, reward, done, info = env.step(action)  #J
 
            agent.push_transition(state, action, reward,
            ➥next_state, done)  #K
            loss = agent.train_step()  #L
            if loss is not None:
                losses_history.append(loss)
 
            episode_reward += reward
            state = next_state
 
            if done:
                success_flag = info["success"]
                break
 
        rewards_history.append(episode_reward)
        success_history.append(1.0 if success_flag else 0.0)
 
        if (episode + 1) % 20 == 0:  #M
            avg_reward = np.mean(rewards_history[-20:])
            avg_success = np.mean(success_history[-20:])
            print(
                f"Episode {episode+1:4d} | "
                f"AvgReward(20): {avg_reward:8.2f} | "f"Success(20):
                ➥{avg_success*100:5.1f}% |
                ➥"f"Epsilon: {agent.epsilon:5.3f} | "
                ➥f"Difficulty: {difficulty:4.2f}")  #N

The training loop updates difficulty before resetting the environment for each episode. section 8.4

How Applied Reinforcement Learning covers it

How deep it goes
Specifies an episode-based difficulty schedule and environment parameters, alongside a success-rate guideline for progression.
Examples
Uses a radiotherapy simulation that adjusts dose noise, tumor dynamics, and organ-dose thresholds over 1,000 episodes.
What you can do after
Operate/configure a simulated training curriculum with episode-based or success-rate-based progression.
How it connects
Connects curriculum learning with deep Q-networks, reinforcement learning environments, and success rates.

Also explained in

Compare how 2 books explain curriculum learning →

This explainer was generated by Manning's knowledge graph from the text of Applied Reinforcement Learning. The section links point to the parts of the book it draws on.