liveBook · Manning

Objective function

As explained in Grokking Machine Learning, Second Edition

A measure of the discrepancy between a regression model's predictions and target values.

An objective function is a number that tells you how well a model is doing. In regression, it measures the gap between predictions and target values: a poor line gets a large number, while a better-fitting line gets a small one. It is also called an error function, loss function, or cost function. section 3.20

Figure 3.17 Two models, a bad one (on the left) and a good one (on the right). The bad one is assigned a large error, and the good one is assigned a small error.

Compare the two fitted lines: the objective function gives the bad fit a higher error and the good fit a lower error. section 3.20

For a line and a set of data points, two useful choices are absolute error and square error. Absolute error adds the vertical distances from the line to the points. Square error adds the squares of those distances. Either gives training a clear goal: make the total as small as possible. section 3.20 section 3.24

An objective function does more than score a finished model:

  • It compares candidate models, assigning lower error to a model that fits the data better. section 3.20
  • It guides gradient descent: repeatedly take a small step that reduces error, like carefully descending a foggy mountain. section 3.24
  • It can help decide when to stop training when error reaches a target or stops decreasing much over several epochs. section 3.25
  • It can balance fit against complexity by adding a regularization term to regression error. section 4.10

Choosing an error for a classifier

Counting wrong classifications is a reasonable start, but it treats every mistake alike. A more useful perceptron error can also reflect how far a misclassified point lies from the decision boundary, so a larger mistake receives a larger penalty. This matters because gradient descent needs an objective that reveals a direction for small improvements rather than one that stays flat. section 5.12 section 5.16

The same training pattern extends to neural networks: start with random weights and biases, define an error function, then repeatedly adjust the parameters to reduce it. For classification, log loss is used because predictions close to their labels receive small values and distant predictions receive large ones. section 10.9 section 10.10

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines several error measures, connects them to gradient descent and stopping, and defers derivative derivations to an appendix.
Examples
Uses competing fitted lines, happy-and-sad classifiers, a mountain descent analogy, and an RMSE training plot.
What it assumes
Introduces minimization through visual intuition without requiring derivatives, while offering derivative-based derivations separately.
What you can do after
Evaluate/understand how an error value compares models, guides parameter updates, and helps choose when training can stop.
How it connects
Connects objective functions to linear regression, gradient descent, regularization, perceptrons, neural networks, and log loss.

Also explained in

Compare how 42 books explain objective function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.