Optimization searches for parameter values that improve a specified objective. In maximum likelihood, the objective is the likelihood—or, commonly, the log-likelihood—to maximize. In machine learning, the objective may instead be a loss to minimize. This turns estimation and model fitting into a shared mathematical task across different models. section 3.1.5
For a neural network, training repeatedly evaluates a loss and adjusts weights and biases using gradients computed by backpropagation. Cross-entropy, for example, measures how closely predicted probabilities match the observed outcome. Many incremental updates reduce the loss, but success on the specified objective does not by itself establish performance on unseen data. section 5.4.1 section 11.2
Optimization can also select among solutions subject to constraints. A hard-margin support vector machine minimizes a function of its weight vector while requiring every training observation to be correctly separated by the margin. Its convex formulation yields a global optimum when the data is linearly separable. Soft margins instead balance margin width against permitted violations; the resulting boundary remains limited by the sample and chosen settings. section 8.2.3 section 10.2.7
Hard-margin SVM objective and constraints: minimize the squared weight norm while requiring correct separation of every training observation. section 8.2.3





































