liveBook · Manning

Objective function

As explained in Grokking Bayes

A loss function is an objective function whose value measures an error or cost and is minimized during optimization.

An objective function turns prediction quality into a number that can guide learning. For a neural network, the loss measures mismatch between predictions and observed data: lower loss means a better fit. section 10.1

A neural network is a function f(x; w), where w contains its weights and biases. Training searches for parameter values that minimize the loss. A common choice is average squared error: calculate each prediction error, square it, and average the results. Gradient descent then moves the parameters toward lower loss. section 10.1

Fitting noisy sine data

A network trained on noisy observations generated from a sine function can achieve a low average squared error and interpolate the observed points well. The loss helps fit the data, but the resulting predictions do not automatically represent observation noise or uncertainty in sparse regions. section 10.1

The same average squared error is also used for linear regression. In Bayesian reasoning, loss can instead describe the consequences of competing decisions, such as deploying or not deploying a website. section 10.1 section 11.0

How Grokking Bayes covers it

How deep it goes
Defines average squared error and parameter minimization, but defers gradient-descent mechanics to separate treatment.
Examples
Uses average squared error with a neural network fitted to noisy sine-function data.
What you can do after
Apply it via gradient descent to fit neural-network parameters using average squared error.
How it connects
Connects neural networks, linear regression, model parameters, gradient descent, predictions, and Bayesian decision-making.

Also explained in

Compare how 42 books explain objective function →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Bayes. The section links point to the parts of the book it draws on.