Imagine studying 80 exam questions and then answering 20 different ones. Those 20 questions resemble unseen data: examples the model did not use during training. Testing on them checks whether the model learned a useful pattern rather than memorized its training set. section 4.2
A simple evaluation procedure separates learning from checking:
- Reserve a small testing set and use the remaining majority as the training set. section 4.2
- Train the model on the training set, then evaluate its predictions on the testing set. section 4.2
- Compare training error with testing error; a large testing error can reveal poor generalization or overfitting. section 4.2
A memorized curve
A degree-10 model may fit every training point yet fail on new points because it memorizes the dataset instead of capturing its downward-opening parabolic pattern. section 4.2
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.
Compare training and testing errors for degree-1, degree-2, and degree-10 models; the degree-2 model has the lowest testing error. section 4.2
