liveBook · Manning

Hyperparameter

As explained in Machines that Think

A hyperparameter is a value selected outside the model's learned parameters to control training or model behavior.

A hyperparameter is chosen outside the model’s learned weights and biases to control how learning proceeds. The learning rate sets the size of each gradient-descent update: large steps can overshoot, while small steps can make progress impractically slow. section 3.2.3

Figure 3.8 The loss landscape can contain many dips and valleys. If the optimizer steps too cautiously, it may settle into a shallow solution with relatively high loss. If it steps too boldly, it may skip over configurations that yield lower loss or bounce back and forth across a good solution without settling. In practice, these landscapes exist in very high-dimensional spaces, where their geometry becomes more complex. Finding the right step size is key to guiding learning toward regions of lower loss.

The loss-landscape illustration contrasts cautious steps that may settle at higher loss with bold steps that may skip or bounce past better configurations. section 3.2.3

How Machines that Think covers it

How deep it goes
Defines hyperparameters through the learning rate and explains how step size affects training, without detailing broader hyperparameter choices.
Examples
Uses a “hot and cold” search analogy and a loss-landscape figure to contrast cautious and overly bold steps.
What you can do after
Understand how learning-rate choices can make training slow, steady, or unstable.
How it connects
Connects hyperparameters to gradient descent, optimization, learning rate, weights and biases, and loss.

Also explained in

Compare how 26 books explain hyperparameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Machines that Think. The section links point to the parts of the book it draws on.