ReduceLROnPlateau adjusts the learning rate from observed validation performance instead of following a fixed epoch schedule. When validation loss stops improving, lowering η can help the model settle into a better solution while avoiding unnecessary reductions during useful progress. section 5.2.4
A practical PyTorch setup follows this pattern:
- Create a validation split rather than relying on training loss, which commonly keeps decreasing under overfitting. section 5.2.4
- Construct the scheduler with the optimizer, a minimizing mode, a reduction factor, and a patience interval. section 5.2.4
- Call step with the latest validation loss after each epoch; unlike ordinary schedules, this scheduler needs the monitored metric. section 5.2.4 section 5.1.3
Minimal configuration
scheduler = torch.optim.lr_scheduler.ReduceLROnPlateau(optimizer, mode='min', factor=0.2, patience=10)\n\n# after each epoch\nscheduler.step(val_running_loss) section 5.2.4 section 5.1.3
fc_model.apply(weight_reset) #1
train_sub_set, val_sub_set = torch.utils.data.random_split(train_data,
[int(len(train_data)*0.8), #2
int(len(train_data)*0.2)])
train_sub_loader = DataLoader(train_sub_set,
batch_size=B, shuffle=True)
val_sub_loader = DataLoader(val_sub_set, batch_size=B) #3
optimizer =torch.optim.SGD(fc_model.parameters(), lr=eta_0)
scheduler =torch.optim.lr_scheduler.ReduceLROnPlateau( #4
optimizer, mode='min', factor=0.2, patience=10)
fc_results_plateau = train_network(fc_model, loss_func, train_loader, #5
val_loader=val_sub_loader, test_loader=test_loader, epochs=epochs,
optimizer=optimizer, lr_schedule=scheduler, score_funcs={'Accuracy':
accuracy_score}, device=device)
#1 Resets the weights again so we don’t have to define a new model
#2 Creates training and validation subsets since we do not have explicit validation and test sets
#3 Creates loaders for the train and validation subsets. Our test loader stays the same—never alter your test data!
#4 Sets up our plateau schedule using gamma=0.2
#5 Train up the model!
Inspect a complete setup showing validation-data creation, optimizer construction, scheduler parameters, and training-loop integration. section 5.2.4
