liveBook · Manning

Evaluation dataset

As explained in Deep Learning with PyTorch, Second Edition

Held-out data used to assess generalization during model development.

An evaluation dataset is data held aside while a model learns from the training set. A common evaluation dataset is the validation set: the optimizer does not use its samples to update parameters, but you can measure validation loss during training. Compare that loss with training loss to see whether the model is learning patterns that carry beyond the examples it fits. section 5.5.3

The two losses answer different questions:

  • If training loss does not fall, the model may be too simple for the data, or the data may lack useful information. section 5.5.3
  • If training loss falls while validation loss rises, the model is overfitting: it fits its training samples without performing as well on independent ones. section 5.5.3
  • If the losses have different values but similar trends, that difference alone does not establish overfitting. section 5.5.3

Make a held-out split

For a small tensor dataset, use torch.randperm to shuffle sample indices, reserve 20% as validation indices, and index both inputs and targets with the same split. In the training loop, calculate both losses but call backward and the optimizer step only on training loss. section 5.5.3

A split should also resemble the inputs the model will encounter in use. For a CT nodule dataset, sorting nodules by size and taking every Nth one for validation helps retain a spread of sizes. A Dataset implementation can use val_stride to designate every tenth sample and isValSet_bool to select training samples, validation samples, or all samples. section 12.2.1 section 12.5.2

Read beyond overall accuracy

A nodule classifier reports about 99.8% validation accuracy, yet identifies none of the positive nodules: it predicts the overwhelmingly common non-nodule class. Inspect results for each class, not just the overall score, especially when missing a positive case matters. section 13.8

Keep a test set separate from both training and validation. Validation results can guide hyperparameter and model-design decisions, so repeated use can overfit that set too. Use the test set only after the model is fully trained for a final independent evaluation. Check for data leakage, including accidental overlap among the splits. section 5.5.4

How Deep Learning with PyTorch, Second Edition covers it

How deep it goes
Works through tensor splitting and separate loss calculations, then shows dataset split controls; does not develop a formal statistical evaluation method.
Examples
Shows a shuffled PyTorch split, a size-spread CT nodule split, and misleading 99.8% validation accuracy in nodule classification.
What it assumes
Assumes basic familiarity with supervised samples and loss while teaching the reason for separating training and validation data.
What you can do after
Diagnose/debug with training and validation losses, split datasets for evaluation, and question high accuracy when positive cases are missed.
How it connects
Connects validation sets to training loss, overfitting, underfitting, hyperparameter selection, test sets, and data leakage.
Cover of Deep Learning with PyTorch, Second Edition
liveBookmanning.com
Read Deep Learning with PyTorch, Second Edition on liveBook Get the book at manning.com

Also explained in

Compare how 27 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with PyTorch, Second Edition. The section links point to the parts of the book it draws on.