An objective function gives model fitting or decision-making a quantity to optimize. A loss function measures error or cost, so minimizing loss selects parameter values that make the model or outcome more desirable. section 2.3.4 section 12.1.1
Maximum likelihood is a special case: maximizing likelihood, or equivalently log-likelihood, becomes minimizing negative log-likelihood. Linear regression can sometimes be solved analytically, whereas neural networks generally require numerical optimization and may settle for a relatively high-likelihood candidate. section 2.3.4
Objective functions can also encode constraints and decision preferences:
- Regularization adds terms to the loss to steer optimization toward preferable parameters; L2 regularization penalizes the sum of squared parameter values and discourages excessively large estimates. section 2.3.4
- In decision models, utility or reward measures desirability, while loss or cost is its negative; optimizing one is equivalent to minimizing the other. section 12.1.1
Investment decision
Investing $1,000 yields utility of $99,000 if the company succeeds and −$1,000 if it fails. The corresponding losses are −$99,000 and $1,000, respectively. Choosing an action requires accounting for uncertainty because success is not guaranteed after investing. section 12.1.1
Figure 12.2 A utility node can represent utility/reward, loss/cost.
A utility node can represent either utility/reward or the corresponding loss/cost within a decision model. section 12.1.1
