An evaluation dataset, often called a validation set, is reserved data used to check a model without training on those examples. Training performance alone can be misleading: a model might memorize its training inputs and targets yet perform poorly on new inputs. Keep validation examples strictly separate from training examples so their loss and metrics remain useful. section 3.6.7
Hold out movie reviews
For an IMDb review classifier, set aside 10,000 examples from the original training data. Train on the remainder for 20 epochs, passing the reserved inputs and targets to Keras through validation_data. The training history then records both training and validation loss and accuracy. In the displayed results, training loss keeps falling across the first five epochs, while validation loss rises from 0.278 to 0.288 between epochs four and five—a reason to watch both curves rather than training loss alone. section 4.1.4
Listing 4.6 Setting aside a validation set
val_indices <- 1:10000 x_val <- x_train[val_indices, ] partial_x_train <- x_train[-val_indices, ] y_val <- y_train[val_indices] partial_y_train <- y_train[-val_indices]
This R listing separates validation inputs and targets from the examples used for fitting. Both inputs and targets must be split using the same indices. section 4.1.4
Use each dataset for a different decision:
- Train on the training set, and monitor validation loss and metrics during fitting with validation_data or validation_split. evaluate() can also compute them afterward. section 3.6.7 section 4.1.4
- Use validation results to choose settings such as model size and number of epochs. Repeated tuning can overfit the validation set even though its examples never update model weights. section 4.1.4 section 5.2.1
- Keep a separate test set untouched by those choices. After selecting a configuration, train a fresh model and use the test set for final evaluation. section 4.1.4 section 5.2.1
A single hold-out split is one option; K-fold cross-validation instead rotates which fold serves as validation data and averages the scores. Choose the split as part of the evaluation protocol, and prevent information leakage. For temporal prediction, validation and test data should occur after training data. section 5.2.1 section 20.1.5
