Generalization is a model’s ability to make accurate predictions on data it has not seen before. A rule that memorizes training examples may fit them perfectly yet fail on new cases. section 1.12 section 4.2
Studying for an exam
Imagine studying 80 of 100 questions and answering the remaining 20 without looking at the answers. The studied questions resemble training data; the unseen questions test whether the learned idea generalizes. Similarly, compare training and testing error rather than trusting training performance alone. section 4.2
Use generalization to judge whether a model has learned a useful pattern:
- Separate a dataset into a training set for learning and a testing set for evaluation. section 4.2
- Watch for overfitting: a highly complex model can reduce training error while increasing testing error. section 4.2
- Prefer the model with low testing error, such as the degree-2 model among the illustrated degree-1, degree-2, and degree-10 choices. section 4.2
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.
Compare how training and testing error change as model complexity increases. section 4.2


















