Training accuracy cannot tell you whether a model will work on new examples. Set aside an evaluation dataset—usually called a validation set—from the training data, then measure its loss and accuracy during training. Use those measurements to choose model size, training duration, and other settings. Keep the test set untouched until the final evaluation: once you make choices based on validation scores, those scores are no longer an unbiased estimate of performance on brand-new data. section 4.1.4 section 20.1.5
Choose when to stop training
For IMDb review classification, hold out 10,000 training examples, train on the remainder for 20 epochs, and pass the held-out inputs and labels as validation_data to model.fit(). Compare the training and validation curves recorded in history.history; keys include loss, accuracy, val_loss, and val_accuracy. After choosing the epoch count, train a new model on the full training set and evaluate it on the test set. section 4.1.4
Listing 4.6 Setting aside a validation set
x_val = x_train[:10000] partial_x_train = x_train[10000:] y_val = y_train[:10000] partial_y_train = y_train[10000:]
The paired slices keep review inputs and labels aligned while reserving the first 10,000 examples for validation. section 4.1.4
A practical decision guide:
- With plenty of data, use a hold-out validation set. In Keras, pass explicit validation_data to fit(), or use validation_split to reserve a fraction of the supplied training arrays. section 4.1.4 section 6.2.2
- With little data, one small hold-out can give a score that changes sharply with the split. Use K-fold cross-validation: train a fresh model for each fold and average its validation scores. For highly accurate evaluation with little data, consider iterated K-fold validation. section 4.3.4 section 6.2.2
- Check that validation examples represent the intended task and are not redundant with training examples. Prevent validation and test labels from leaking into training; for temporal prediction, place both sets after the training period. section 6.2.2 section 20.1.5
When a single split is too noisy
In a small house-price regression task, four folds produced validation scores from 0.232 to 0.349; their mean, 0.296, was more reliable than any one score. Tracking mean validation error by epoch then supported a final training run of 130 epochs before test evaluation. section 4.3.4
Treat validation results as guidance, not as a score to optimize indefinitely. Repeated hyperparameter tuning can overspecialize choices to the validation set; the separate test set remains the final check. section 20.1.5
