liveBook · Manning

Hyperparameter

As explained in Grokking Machine Learning, Second Edition

A quantity that is set before the training process and controls how a model is trained.

A hyperparameter is a quantity set before training that controls how a model learns. A model creates or changes its parameters during training; hyperparameters are the knobs chosen beforehand. section 3.31

A perceptron’s training settings

A perceptron uses the learning rate to control how far its line moves after a mistake and the number of epochs to control how many times it repeats training. Both are hyperparameters; the resulting weights and bias are parameters. section 5.15

Common hyperparameters control different parts of training:

  • Learning rate: the size of each training step. section 5.15 section 10.16
  • Number of epochs: how many times the training procedure runs. section 5.15 section 10.16
  • Decision-tree settings: minimum samples for splitting or leaves, minimum improvement, and maximum tree depth. section 9.5
  • Neural-network settings: batch size, layer and node counts, activation functions, regularization, and dropout probability. section 10.16

Hyperparameters often determine when training stops, how quickly it proceeds, and how much data enters each step. Poor choices can allow a decision tree to split too far and overfit. A practical way to choose among combinations is grid search: test combinations and keep the one performing best on a validation set. section 9.5 section 10.16

How Grokking Machine Learning, Second Edition covers it

How deep it goes
Defines the parameter distinction and lists tuning examples, stopping short of detailed search procedures or optimization internals.
Examples
Uses perceptron learning rate and epochs, tree depth and sample limits, and neural-network architecture, regularization, and dropout.
What it assumes
Assumes basic awareness of model training while defining the distinction between parameters and hyperparameters directly.
What you can do after
Evaluate training settings and choose among hyperparameter combinations using performance on a validation set.
How it connects
Connects hyperparameters with model parameters, learning rate, epochs, decision trees, neural networks, overfitting, validation sets, and grid search.

Also explained in

Compare how 26 books explain hyperparameter →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking Machine Learning, Second Edition. The section links point to the parts of the book it draws on.