liveBook · Manning

Optimization

As explained in Timeless Algorithms, The Foundational Ideas

An optimization method or algorithm is a procedure for searching for solutions that improve an objective function.

Optimization searches for parameter values that improve a specified objective. In maximum likelihood, the objective is the likelihood—or, commonly, the log-likelihood—to maximize. In machine learning, the objective may instead be a loss to minimize. This turns estimation and model fitting into a shared mathematical task across different models. section 3.1.5

For a neural network, training repeatedly evaluates a loss and adjusts weights and biases using gradients computed by backpropagation. Cross-entropy, for example, measures how closely predicted probabilities match the observed outcome. Many incremental updates reduce the loss, but success on the specified objective does not by itself establish performance on unseen data. section 5.4.1 section 11.2

Optimization can also select among solutions subject to constraints. A hard-margin support vector machine minimizes a function of its weight vector while requiring every training observation to be correctly separated by the margin. Its convex formulation yields a global optimum when the data is linearly separable. Soft margins instead balance margin width against permitted violations; the resulting boundary remains limited by the sample and chosen settings. section 8.2.3 section 10.2.7

Minimize 12||w||2 subject to yi(w∙x+b)≥1 for all i

Hard-margin SVM objective and constraints: minimize the squared weight norm while requiring correct separation of every training observation. section 8.2.3

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Connects objectives to iterative updates and constrained formulations, but distinguishes an objective's optimum from future performance.
Examples
Uses handwritten-digit classification with cross-entropy and maximum-margin separation with a hard-margin SVM.
What you can do after
Apply it to maximize likelihood, reduce neural-network loss through gradient updates, or select a margin under stated constraints.
How it connects
Links maximum likelihood, loss functions, cross-entropy, gradients, neural networks, and support vector machines.

Also explained in

Compare how 38 books explain optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.