An objective function gives an algorithm a quantity to optimize. In prediction tasks, the loss function—also called a cost function or objective function—compares predicted values with actual values. A larger loss means a greater discrepancy; training aims to reduce it. For example, a demand forecast can be judged by how far its predictions are from observed demand. section 6.2.2 section 8.3.3
For a neural network, write the comparison as C(y, ŷ), where y is the expected output and ŷ is the network’s output. Backpropagation calculates how C changes with respect to each weight, working backward through the layers using the chain rule. An optimizer then adjusts weights and biases to reduce loss over successive iterations. Batch gradient descent uses the training dataset for each iteration; stochastic gradient descent updates after each instance; mini-batches use small subsets and require a batch-size choice. section 8.4.1 section 8.2.6
An objective need not compare forecast values. In multidimensional scaling, the cost function is called stress: reducing it helps represent higher-dimensional distances in fewer dimensions. In fuzzy c-means clustering, minimizing the cost function is the stopping criterion for the iterative clustering process. Thus, interpret a low value in terms of the particular objective being optimized, not as a universal accuracy score. section 6.2.2 section 5.5.1
Table 6.2 Stress values and their goodness of fit
|
Stress values
|
Goodness of fit
|
|---|---|
|
0.200
|
Poor
|
|
0.100
|
Fair
|
|
0.050
|
Good
|
|
0.025
|
Excellent
|
|
0.000
|
Perfect
|
The stress table gives concrete goodness-of-fit labels, from 0.200 (poor) to 0.000 (perfect), for interpreting this particular objective. section 6.2.2
