How well might a model predict new examples? Test error measures its prediction errors on a testing set held out from training. Compare it with training error: a model that fits training data well but has high test error may be overfitting. section 4.2
For a regression example, mean absolute error (MAE) averages the lengths of the vertical gaps between data points and the model. Comparing test errors across models helps identify one that fits well without simply memorizing the training data. section 4.2
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.
Compare training and testing errors for three polynomial models; the degree-2 model has the lowest testing error. section 4.2
Testing error can also be reported as root mean square error (RMSE). In a polynomial regression example, the models with L1 and L2 regularization have lower testing RMSE than the unregularized model. section 4.6
