liveBook · Manning

Objective function

As explained in Timeless Algorithms, The Foundational Ideas

A training objective is a quantity that a learning procedure seeks to optimize while fitting a model.

Fitting a model requires a criterion for deciding whether its current parameters are better. An objective function supplies that criterion: it is a quantity the learning procedure seeks to minimize or maximize. section 11.2 section 8.2.3

A loss function measures the discrepancy between a prediction and an observed outcome. Smaller loss means closer agreement, but classification loss retains information that a thresholded correct-or-incorrect result discards: cross-entropy, for example, penalizes confident errors more heavily than uncertain ones. section 11.2.3 section 11.8

The objective depends on the task and formulation:

  • Use squared-error loss for regression and cross-entropy loss for classification. section 11.2.3
  • For a hard-margin support vector machine, minimize one-half the squared weight-vector norm subject to constraints that correctly separate the training observations. section 8.2.3

In neural-network training, backpropagation calculates how each parameter affects the loss, and an optimization algorithm uses those gradients to update parameters incrementally. For language models, cross-entropy on held-out text measures how much probability the model assigns to the observed next tokens. section 11.2 section 13.1.2

Minimize 12||w||2 subject to yi(w∙x+b)≥1 for all i

The hard-margin SVM objective makes the distinction between what is optimized and what solutions are allowed explicit: the norm is minimized subject to separation constraints. section 8.2.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Explains loss behavior and gradient-based updates, then formulates hard-margin SVM optimization; soft-margin treatment is deferred.
Examples
Compares squared-error regression, cross-entropy classification, hard-margin SVM optimization, and next-token prediction measured on held-out text.
What it assumes
Assumes predictions, observed outcomes, and model parameters, while defining how prediction discrepancies become a learning signal.
What you can do after
Evaluate how a chosen objective scores predictions or constrains model fitting, and distinguish continuous loss from thresholded classification.
How it connects
Connects to backpropagation, gradient descent, cross-entropy, support vector machines, maximum likelihood, and information theory.

Also explained in

Compare how 42 books explain objective function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.