An objective function assigns a value to a candidate solution so that solutions can be compared. It might measure profit to maximize, cost to minimize, or—when training a predictive model—loss to minimize. Decision variables determine its inputs; constraints determine which solutions are allowed. The best objective value among feasible solutions is optimal, while a good but not necessarily best value is near-optimal. section 1.3.2 section 1.3
Choose a ticket price
Suppose a conference sells Q = 5,000 − 20x tickets at price x. With fixed cost a and cost b per attendee, profit is f(x) = Qx − (a + Qb). Set a = 50,000, b = 60, and restrict x to $0–$250. The resulting function is f(x) = −20x² + 6,200x − 350,000. Setting its derivative to zero gives x = $155 and profit of $130,500. Here the price is the decision variable, profit is the objective, and the price bounds are constraints. section 1.3.1 section 1.3.2
The choice of objective changes what a search regards as good:
- One objective can be maximized or minimized. Multiple objectives can express competing aims, such as reducing an electric vehicle’s acceleration time while increasing its driving range. section 1.3.2
- A soft constraint can become a reward or penalty within the objective. A cargo-bike loading example penalizes exceeding its weight capacity; the penalty can change as search progresses. section 1.3.3
- In neural-network training, loss measures the gap between predictions and targets. Mean squared error is used for regression; cross-entropy and negative log likelihood are described for classification. section 9.7
An optimizer evaluates candidate decision-variable values and seeks better feasible ones. A black-box solver needs objective and constraint values for those candidates, without requiring the functions to be differentiable. Yet an attractive local minimum may not be the global minimum. Simulated annealing can accept a worse candidate with a probability determined by the objective-value change and temperature, helping it move beyond some local minima. section 1.4.3 section 5.2.3
def forward_prop(params):
logits = logits_function(params) #1
exp_scores = np.exp(logits) #2
probs = exp_scores / np.sum(exp_scores, axis=1, keepdims=True) #2
correct_logprobs = -np.log(probs[range(num_samples), y]) #3
loss = np.sum(correct_logprobs) / num_samples #4
return loss #4
The Python forward pass turns class probabilities into one scalar loss: take the negative log probability assigned to each correct label, then average across samples. That scalar lets the optimizer score a proposed set of network parameters. section 9.7
Before optimizing, ask whether the number captures what matters. Elevator dispatch, for example, could prioritize average waiting time, total time to destination, unusually long waits, or fairness. Incoming requests can change what looks best immediately, making a fixed notion of optimum difficult in a dynamic, partly observable problem. section 1.4.2
