Data leakage is the unintended transfer of evaluation information into a model or its development decisions. Repeatedly tuning hyperparameters against validation performance gradually teaches the development process about the validation set, making its score artificially optimistic. A separate test set must remain untouched until final evaluation so it can measure performance on genuinely new data. section 5.2.1
Prevent leakage with a disciplined evaluation workflow:
- Reserve training data for fitting model parameters. section 5.2.1
- Use validation data to compare configurations, recognizing that repeated experiments can overfit it. section 5.2.1
- Keep a completely separate test set for one final estimate of generalization. section 5.2.1 section 5.5
- When data is scarce, estimate validation performance with K-fold or iterated K-fold validation, then retrain on all available development data before testing. section 5.2.1
Listing 5.5 Hold-out validation (labels omitted for simplicity)
num_validation_samples <- 10000 val_indices <- sample.int(nrow(data), num_validation_samples) #1 validation_data <- data[val_indices, ] #2 training_data <- data[-val_indices, ] #3 model <- get_model() #4 fit(model, training_data, ...) validation_score <- evaluate(model, validation_data, ...) ... #5 model <- get_model() #6 fit(model, data, ...) #6 test_score <- evaluate(model, test_data, ...) #6
Shows a complete hold-out workflow: split data, fit on training data, evaluate on validation data, then retrain and evaluate once on the test set. section 5.2.1
