liveBook · Manning

Learning rate

As explained in Build AI Drug Discovery Pipelines

A learning rate is an optimization step-size configuration that controls the magnitude of parameter updates during iterative learning.

A learning rate controls the size of updates during iterative learning. In gradient descent, the gradient measures how changes to model weights affect the cost; the weights move in the negative gradient direction, scaled by the learning rate. section 4.2.3 section 8.2.1

Think of descending a hilly landscape by feeling the slope and stepping downhill. The update depends on both the learning rate and the gradient: even with a fixed rate, a smaller gradient produces a smaller step. A rate that is too large can make learning unstable; stochastic gradient descent can also oscillate near a minimum because its updates are noisy. section 4.2.3 section 5.1.1

Figure 4.11 Impact of learning step size on the ability for gradient descent to converge.

Compare how learning step size affects convergence. section 4.2.3

A learning rate scheduler adjusts the rate during training. To guide changes, track training and validation loss over epochs. If validation loss levels off, consider adjusting the rate or scheduler, reducing the number of epochs, or using early stopping. section 8.2.3

Solubility regression

A solubility model uses scikit-learn's SGDRegressor with an inverse-scaling learning rate, early stopping, and epsilon-insensitive loss. The grid search also tests adaptive and constant learning rates. The selected model's test RMSE is 1.2783, compared with 1.3787 for the ESOL baseline and 2.4585 for a y-scrambled baseline. section 4.3.3

Binding-affinity surrogate

A neural-network surrogate for docking uses PyTorch's Adam optimizer with a starting learning rate of 0.001 and a default batch size of 32. The rate is a practical starting assumption, not a guaranteed optimum; it can be tuned while monitoring convergence and comparing experiments. section 9.2.3

Listing 9.6 Training the Deep Learning Surrogate Model
def train_deep_model(
    model, X_train, y_train, device, writer=None, global_step=0, 
    epochs=10, batch_size=32, lr=0.001, **kwargs
):
    history = {"train_loss": []}
    
    criterion = nn.MSELoss()    #A
    optimizer = torch.optim.Adam(model.parameters(), lr=lr)
    
    X_train_tensor = torch.tensor(
        X_train, dtype=torch.float32
    ).to(device)    #B
    y_train_tensor = torch.tensor(
        y_train, dtype=torch.float32
    ).view(-1, 1).to(device)
    
    train_data = torch.utils.data.TensorDataset(
        X_train_tensor, y_train_tensor
    )    #C
    train_loader = torch.utils.data.DataLoader(
        train_data, batch_size=batch_size, shuffle=True
    )
    
    for epoch in range(epochs):
        model.train()
        epoch_loss = 0.0
        num_batches = 0
        
        for inputs, labels in train_loader:
            optimizer.zero_grad()    #D
            outputs = model(inputs)    #E
            loss = criterion(outputs, labels)    #F
            loss.backward()    #G
            optimizer.step()
            epoch_loss += loss.item()    #H
            num_batches += 1
        
        avg_epoch_loss = epoch_loss / num_batches
        history["train_loss"].append(avg_epoch_loss)
        if writer is not None:    #I
            writer.add_scalar(
                'Training/Loss', avg_epoch_loss,
                global_step*epochs + epoch
            )
            
        print(f"Epoch {epoch + 1}/{epochs}, Loss: {avg_epoch_loss:.4f}")
    return history

See where the learning rate is passed to Adam in the surrogate training function. section 9.2.3

In gradient boosting, the learning rate has a related role: it scales each new tree's contribution to the ensemble, a practice called shrinkage. A lower rate may require more trees to fit the training set. Tune the rate alongside tree count and depth, and use early stopping to help choose the number of trees. section 6.4.2

How Build AI Drug Discovery Pipelines covers it

How deep it goes
Explains gradient-scaled updates, compares scheduler choices, and describes the tradeoff between boosting learning rate and tree count.
Examples
Illustrates rate selection with solubility regression and an Adam-trained binding-affinity surrogate for docking.
What you can do after
Apply learning-rate settings with scikit-learn's SGDRegressor and PyTorch's Adam, and monitor losses to inform tuning.
How it connects
Connects learning rate to gradient descent, stochastic gradient descent, early stopping, and gradient boosting.

Also explained in

Compare how 23 books explain learning rate →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build AI Drug Discovery Pipelines. The section links point to the parts of the book it draws on.