Testing evaluates a trained model on data held aside from training. Comparing its predictions with the test data helps check whether the model generalizes beyond examples it has already seen. section 4.2
Exam questions and model complexity
Imagine studying 80 of 100 exam questions and using the remaining 20 to test yourself. For a dataset, compare models of degrees 1, 2, and 10: training error falls as complexity rises, but testing error first falls and then rises. The degree-2 model has the lowest testing error in this example. section 4.2
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.
Compare the training and testing errors for three model complexities; the figure shows why the lowest training error does not necessarily identify the best model. section 4.2
