liveBook · Manning

Learning rate schedule

As explained in Build AI Drug Discovery Pipelines

Learning rate scheduling is the adjustment of an optimizer's learning rate during model training according to a predefined or adaptive schedule.

A learning rate schedule adjusts an optimizer’s learning rate during neural-network training. In a PyTorch training function, the scheduler is supplied alongside the model, data loaders, loss function, optimizer, device, and epoch count. section 8.2.3

Use the schedule within an iterative training and validation workflow:

  • Process training batches by zeroing gradients, running a forward pass, calculating loss, backpropagating, and updating parameters with the optimizer. section 8.2.3
  • Track average training loss and evaluate validation loss after each epoch to monitor whether further training remains useful. section 8.2.3
  • Treat the scheduler as a tunable training control when validation loss saturates, alongside reducing epochs, using early stopping, or changing the learning rate. section 8.2.3

Drug-target affinity training

A graph neural network for predicting drug-target affinity uses Adam with learning rate 0.001 and no learning rate scheduling. In a separate training run, validation loss improves early but fluctuates and saturates before epoch 50, motivating scheduler or early-stopping experiments. section 11.5.6 section 8.2.3

Listing 8.7 Train the Model
def train_model(
    model, train_loader, val_loader, criterion, optimizer, 
    scheduler, device, num_epochs=10
):
    model.train()    #A
    train_losses = []    #B
    val_losses = []
    
    for epoch in range(num_epochs):    #C
        running_loss = 0.0    #D
        for inputs, targets in train_loader:
            inputs, targets = inputs.to(device), targets.to(device)
            
            optimizer.zero_grad()    #E
            outputs = model(inputs)    #F
            loss = criterion(outputs, targets)    #G
            loss.backward()    #H
            optimizer.step()    #I
            
            running_loss += loss.item()
        
        train_loss = running_loss / len(train_loader)
        train_losses.append(train_loss)
        
        model.eval()    #J
        val_loss = 0.0
        with torch.no_grad():    #K
            for inputs, targets in val_loader:
                inputs, targets = inputs.to(device), targets.to(device)
                outputs = model(inputs)
                loss = criterion(outputs, targets)
                val_loss += loss.item()
        
        val_loss /= len(val_loader)
        val_losses.append(val_loss)
        
        print(f'Epoch {epoch+1}/{num_epochs}', 
              f'Train Loss: {train_loss:.4f}',
              f'Val Loss: {val_loss:.4f}')    #L
        
        scheduler.step(val_loss)    #M
        model.train()    #N

Inspect the PyTorch training function that accepts a scheduler with the optimizer and records training and validation losses across epochs. section 8.2.3

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains scheduler integration and tuning rationale but does not specify particular scheduling policies or step rules.
Examples
Uses a PyTorch training listing, epoch-wise loss logs, and a drug-target affinity model trained with Adam without scheduling.
What it assumes
Assumes familiarity with model training, optimizers, loss functions, epochs, batches, and validation data.
What you can do after
Diagnose validation-loss saturation and apply scheduler changes as part of neural-network hyperparameter tuning.
How it connects
Connects learning rate scheduling with optimizers, backpropagation, validation loss, early stopping, and hyperparameter tuning.

Also explained in

Compare how 4 books explain learning rate schedule →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.