Imagine studying 80 questions from a 100-question exam, then answering 20 unseen questions. A training dataset works similarly: it is the portion used to fit a machine-learning model. section 4.2
Use the dataset portions for different jobs:
- Train every candidate model with the training dataset. section 4.3
- Use a validation set to choose a model or its hyperparameters, rather than reusing testing data. section 4.3
- Reserve the testing set for the final evaluation of generalization to unseen data. section 4.7
Splitting a real dataset
A Titanic classification dataset can be divided into 60% training, 20% validation, and 20% testing data. Features train models; labels provide the outcomes to learn. With Scikit-Learn, splitting can happen twice, producing separate training, validation, and testing tables before fitting several classifiers. section 13.4
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.
Compare training and testing errors for models of degrees 1, 2, and 10; increasing complexity lowers training error but can raise testing error. section 4.2
