Fitting a model requires a criterion for deciding whether its current parameters are better. An objective function supplies that criterion: it is a quantity the learning procedure seeks to minimize or maximize. section 11.2 section 8.2.3
A loss function measures the discrepancy between a prediction and an observed outcome. Smaller loss means closer agreement, but classification loss retains information that a thresholded correct-or-incorrect result discards: cross-entropy, for example, penalizes confident errors more heavily than uncertain ones. section 11.2.3 section 11.8
The objective depends on the task and formulation:
- Use squared-error loss for regression and cross-entropy loss for classification. section 11.2.3
- For a hard-margin support vector machine, minimize one-half the squared weight-vector norm subject to constraints that correctly separate the training observations. section 8.2.3
In neural-network training, backpropagation calculates how each parameter affects the loss, and an optimization algorithm uses those gradients to update parameters incrementally. For language models, cross-entropy on held-out text measures how much probability the model assigns to the observed next tokens. section 11.2 section 13.1.2
The hard-margin SVM objective makes the distinction between what is optimized and what solutions are allowed explicit: the norm is minimized subject to separation constraints. section 8.2.3









































