The golden rule protects honest model evaluation: use training data to train the model, and never use testing data for training or for decisions about model hyperparameters. section 4.2
Breaking the rule can cause overfitting that remains invisible during development. A model may look excellent yet fail on a secret dataset, especially when testing data was used accidentally. section 4.2
Choosing model complexity
Compare models using testing error without training on the testing set. For degree-1, degree-2, and degree-10 models, testing error first falls and then rises; the degree-2 model is preferred because it has low testing error. section 4.2
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.
Use the figure to compare training and testing errors as model complexity changes. section 4.2
