A learning rate controls the size of updates during iterative learning. In gradient descent, the gradient measures how changes to model weights affect the cost; the weights move in the negative gradient direction, scaled by the learning rate. section 4.2.3 section 8.2.1
Think of descending a hilly landscape by feeling the slope and stepping downhill. The update depends on both the learning rate and the gradient: even with a fixed rate, a smaller gradient produces a smaller step. A rate that is too large can make learning unstable; stochastic gradient descent can also oscillate near a minimum because its updates are noisy. section 4.2.3 section 5.1.1
Figure 4.11 Impact of learning step size on the ability for gradient descent to converge.

Compare how learning step size affects convergence. section 4.2.3
A learning rate scheduler adjusts the rate during training. To guide changes, track training and validation loss over epochs. If validation loss levels off, consider adjusting the rate or scheduler, reducing the number of epochs, or using early stopping. section 8.2.3
Solubility regression
A solubility model uses scikit-learn's SGDRegressor with an inverse-scaling learning rate, early stopping, and epsilon-insensitive loss. The grid search also tests adaptive and constant learning rates. The selected model's test RMSE is 1.2783, compared with 1.3787 for the ESOL baseline and 2.4585 for a y-scrambled baseline. section 4.3.3
Binding-affinity surrogate
A neural-network surrogate for docking uses PyTorch's Adam optimizer with a starting learning rate of 0.001 and a default batch size of 32. The rate is a practical starting assumption, not a guaranteed optimum; it can be tuned while monitoring convergence and comparing experiments. section 9.2.3
Listing 9.6 Training the Deep Learning Surrogate Model
def train_deep_model(
model, X_train, y_train, device, writer=None, global_step=0,
epochs=10, batch_size=32, lr=0.001, **kwargs
):
history = {"train_loss": []}
criterion = nn.MSELoss() #A
optimizer = torch.optim.Adam(model.parameters(), lr=lr)
X_train_tensor = torch.tensor(
X_train, dtype=torch.float32
).to(device) #B
y_train_tensor = torch.tensor(
y_train, dtype=torch.float32
).view(-1, 1).to(device)
train_data = torch.utils.data.TensorDataset(
X_train_tensor, y_train_tensor
) #C
train_loader = torch.utils.data.DataLoader(
train_data, batch_size=batch_size, shuffle=True
)
for epoch in range(epochs):
model.train()
epoch_loss = 0.0
num_batches = 0
for inputs, labels in train_loader:
optimizer.zero_grad() #D
outputs = model(inputs) #E
loss = criterion(outputs, labels) #F
loss.backward() #G
optimizer.step()
epoch_loss += loss.item() #H
num_batches += 1
avg_epoch_loss = epoch_loss / num_batches
history["train_loss"].append(avg_epoch_loss)
if writer is not None: #I
writer.add_scalar(
'Training/Loss', avg_epoch_loss,
global_step*epochs + epoch
)
print(f"Epoch {epoch + 1}/{epochs}, Loss: {avg_epoch_loss:.4f}")
return historySee where the learning rate is passed to Adam in the surrogate training function. section 9.2.3
In gradient boosting, the learning rate has a related role: it scales each new tree's contribution to the ensemble, a practice called shrinkage. A lower rate may require more trees to fit the training set. Tune the rate alongside tree count and depth, and use early stopping to help choose the number of trees. section 6.4.2






















