liveBook · Manning

Optimization

As explained in Machines that Think

Optimization is the process of adjusting a model's parameters according to specified rules to reduce error or improve an objective.

In neural-network training, optimization turns information about error into updates to weights and biases, aiming to move future predictions closer to the desired outcome. More broadly, statistical optimization can shape model behavior under competing objectives without implying human-like intention. section 3.2.3 section 7.2.4

The update process involves several linked ideas:

  • Gradient descent moves weights and biases opposite the gradient, which points toward increasing loss. section 3.2.3
  • The learning rate controls step size. Large steps can overshoot; small ones can make progress impractically slow. section 3.2.3
  • Momentum carries part of earlier movement forward, smoothing noisy gradient signals and supporting steadier progress. section 3.2.3

Finding a hidden object

The “hot and cold” analogy captures step-size tradeoffs: small moves may be slow, while large moves may overshoot the target. section 3.2.3

When models have tools, autonomy, and performance objectives, optimization can affect real-world decisions. Ethical constraints remain part of predictive calculations rather than categorical limits, so oversight and guardrails matter alongside alignment. section 7.2.4

Figure 3.8 The loss landscape can contain many dips and valleys. If the optimizer steps too cautiously, it may settle into a shallow solution with relatively high loss. If it steps too boldly, it may skip over configurations that yield lower loss or bounce back and forth across a good solution without settling. In practice, these landscapes exist in very high-dimensional spaces, where their geometry becomes more complex. Finding the right step size is key to guiding learning toward regions of lower loss.

The loss-landscape figure illustrates how cautious or bold steps can lead to shallow solutions, missed low-loss configurations, or unstable bouncing. section 3.2.3

How Machines that Think covers it

How deep it goes
Explains gradient descent, learning-rate tradeoffs, and momentum qualitatively, then connects optimization to action under competing objectives.
Examples
Uses a hot-and-cold search analogy, a loss-landscape figure, and examples of actions such as trades and resource allocation.
What it assumes
Assumes the gradient indicates how weights affect error and distinguishes learned weights and biases from the researcher-chosen learning rate.
What you can do after
Understand how update rules trade speed against stability and interpret goal-directed language as optimization under pressure, not proof of motives.
How it connects
Connects gradient descent, gradients, loss, learning rate, and momentum to training, and links statistical optimization to alignment.

Also explained in

Compare how 38 books explain optimization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.