An evaluation dataset, often called a validation set, is held apart from the data used to update a model’s parameters. Its loss helps you choose hyperparameters and decide when to stop training. A separate test set assesses the trained model afterward. This separation matters because a model can perform well on its training data but poorly on unseen data. section 2.2.1 section 2.4.1
For a Fashion MNIST clothing classifier, the workflow is:
- Split 60,000 training observations into 50,000 for training and 10,000 for validation, then make separate PyTorch DataLoader iterators. section 2.4.1
- After each training epoch, predict on validation batches and calculate validation loss without updating model parameters. section 2.4.2
- Track the lowest validation loss. Stop when it has not improved for 10 epochs; in this run, training stops after 19 epochs. section 2.4.1 section 2.4.2
def val_epoch():
vloss=0
for n,(imgs,labels) in enumerate(val_loader):
imgs=imgs.reshape(-1,28*28).to(device)
labels=labels.reshape(-1,).to(device)
preds=model(imgs)
loss=loss_fn(preds,labels)
vloss+=loss.detach()
return vloss/n
The validation loop computes loss from predictions but contains no backward pass or optimizer step, unlike the training loop. section 2.4.2
Listing 2.8 The EarlyStop() class to determine when to stop training
class EarlyStop:
def __init__(self, patience=10): #1
self.patience = patience
self.steps = 0
self.min_loss = float('inf')
def stop(self, val_loss): #2
if val_loss < self.min_loss: #3
self.min_loss = val_loss
self.steps = 0
elif val_loss >= self.min_loss: #4
self.steps += 1
if self.steps >= self.patience:
return True
else:
return False
stopper=EarlyStop()
EarlyStop records the best validation loss and counts epochs without improvement until its patience threshold is reached. section 2.4.1
Keep the final check separate
Once the clothing classifier has stopped, evaluate it on the test set rather than using that set to choose the stopping point. Its reported test accuracy is about 0.882. The same three-way distinction applies beyond images: MAESTRO piano MIDI files are organized into 967 training, 137 validation, and 178 test files. section 2.4.2 section 14.2.1
