An evaluation dataset is a reserved subset of observations used to measure how well a fitted model generalizes to unseen data. Keeping it separate from training helps detect overfitting and supports more reliable performance assessment. section 6.3.1 section 6.5
Use an evaluation dataset to:
- Assess predictions on observations the model has not seen during fitting. section 6.3.1
- Compare predicted labels with true labels using accuracy and a confusion matrix. section 6.3.4
- Preserve chronological order when forecasting by testing on observations that follow the training period. section 7.3.7
A reproducible classification split
For fourth-down conversion prediction, select features X and target y, then call train_test_split(X, y, test_size=0.3, random_state=0). The resulting training data contains 618 rows and the evaluation data contains 265 rows. A random forest predicts y_test from X_test, and accuracy compares those predictions with the held-out labels. section 6.3.1 section 6.4.2 section 6.3.4
A chronological forecasting split
For Apple closing prices, fit the forecasting model on October 1, 2023 through March 31, 2024, then compare predictions for the remaining 22 April trading days with their actual prices. Apply stationarity checks and preprocessing to the training subset. section 7.3.7
>>> X = nfl_pbp[['QUARTER', 'TO_GO', 'OFFENSIVE_TEAM_VENUE', >>> 'SCORE_DIFFERENTIAL', 'PLAY_TYPE']]
Inspect the feature selection used to construct the input matrix before separating training and evaluation data. section 6.3.1
