liveBook · Manning

Hyperparameter

As explained in Timeless Algorithms, The Foundational Ideas

A hyperparameter is a setting that controls a model or learning procedure and is selected outside ordinary parameter fitting.

A hyperparameter is a setting chosen outside ordinary parameter fitting. Gradient descent updates weights and biases, but settings such as learning rate and network depth are selected separately. section 11.2.5 section 11.6.1

Common choices include:

  • Architecture: depth, width, activation functions, and connectivity. section 11.6.1
  • Training: learning rate, batch size, and regularization strength. section 11.6.1
  • Tree models: number of trees or decision-tree depth. section 2.4.1
θnew=θold-η∂L∂θ

The learning rate η sets the size of each gradient-descent adjustment; a value that is too large can overshoot, while one that is too small slows training. section 11.2.5

Selection is empirical: train candidate configurations and compare them on validation data, then evaluate the selected configuration on reserved test data. Bayesian optimization can use results from earlier evaluations to choose promising settings, rather than testing every combination. section 11.6.1 section 2.4.1

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines settings, explains effects on capacity and training, and describes validation-based selection without guaranteeing a globally best configuration.
Examples
Names depth, width, activation functions, learning rate, batch size, regularization strength, tree count, and decision-tree depth.
What you can do after
Evaluate candidate configurations on validation data, then reserve test data for final evaluation.
How it connects
Links hyperparameter selection to validation data and Bayesian optimization.

Also explained in

Compare how 26 books explain hyperparameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.