How can you tell whether a line fits a set of points? An objective function assigns the model a number: a large error for a poor fit and a small error for a good one. When that number measures prediction mistakes on the data used for training, it is training error. Objective functions that measure error are also called error functions, loss functions, or cost functions. section 3.4 section 4.2
For linear regression, look at the vertical gap between each point and the line’s prediction. Absolute error adds the sizes of those gaps; square error adds their squares. The mean absolute error averages the absolute gaps instead. Each turns the visual question “How close is the line?” into a number you can compare across models. section 3.4 section 4.2
The choice of objective depends on what counts as a useful mistake. For a classifier, counting wrong predictions distinguishes eight mistakes from three, but says nothing about how severe each mistake is. A misclassified point far across a decision boundary can be treated differently from one just across it. An objective used with gradient descent must also provide a direction in which a small step can reduce error; a flat error function does not. section 5.2
Training then becomes a repeated attempt to lower the chosen objective. Picture each possible model as a location on an error mountain: take small downhill steps toward models with lower error. For a neural network, this means starting with random weights and biases, calculating the loss and its gradient, and updating the weights and biases in the opposite direction. Log loss is one objective used there for classification: it is small when a prediction is close to its label and large when it is far away. section 3.4 section 10.2
But should you always choose the model with the smallest training error? No. A model can memorize its training points and predict new points poorly. Compare training error with error on held-out data: high error on both suggests underfitting, while low training error paired with high testing or validation error suggests overfitting. In a model complexity graph, the useful choice is the model with the lowest validation error among those considered, not necessarily the one with the lowest training error. section 4.2 section 4.4
Figure 4.4 We can use this table to decide how complex we want our model. The columns represent the three models of degree 1, 2, and 10. The rows represent the training and the testing error. The solid circles are the training set, and the white triangles are the testing set. The errors at each point can be seen as the vertical lines from the point to the curve. The error of each model is the mean absolute error given by the average of these vertical lengths. Notice that the training error goes down as the complexity of the model increases. However, the testing error goes down and then back up as the complexity increases. From this table, we conclude that out of these three models, the best one is model 2, because it gives us a low testing error.
Compare the three polynomial models row by row: the most complex reaches zero training error, yet a simpler model has the lower testing error. section 4.2
Sometimes prediction error is only part of the objective. Regularization adds a complexity term to regression error: error = regression error + λ × regularization term. An L1 term gives lasso regression; an L2 term gives ridge regression. The goal is a model that fits without needless complexity, rather than one that merely drives training error down. section 4.5
Try choosing a model
Four models have training/testing errors of 0.1/1.8, 0.4/1.2, 0.6/0.8, and 1.9/2.3. Which would you select? The third has the lowest testing error, even though its training error is not the lowest. Use the paired errors to identify a likely overfit and a likely underfit model. section 4.8
