Before training a model, you can choose how it will learn or what form it can take. Those choices are hyperparameters. A useful rule is: if you set a quantity before training, it is a hyperparameter; if the model creates or changes it during training, it is a parameter. In regression, for example, the learning rate and number of epochs are hyperparameters, while the learned weights and bias are parameters. section 3.7
Different models offer different knobs to set:
- For a decision tree, maximum depth and minimum samples per leaf are stopping-rule hyperparameters. They can prevent a tree from splitting until its leaves contain very few examples, which risks overfitting. section 9.2
- For a neural network, choices include learning rate, number of epochs, batch size, layers, nodes per layer, activation functions, regularization settings, and dropout probability. section 10.2
You do not have to settle on the first settings you try. Grid search trains several models with different hyperparameter choices. In a practical workflow, split data into training, validation, and testing portions before training and evaluating models; k-fold cross-validation can reuse different portions for training and validation. section 10.2 section 13.7
Try changing a stopping rule
Imagine a decision tree recommending an app from a user’s age and phone platform. Try two maximum depths. For each, note whether the tree stops before every leaf has a single label, then compare the models during validation. The point is not simply to fit every original example: splitting too far can cause overfitting. section 9.2 section 13.7
