A hyperparameter is a setting chosen outside ordinary parameter fitting. Gradient descent updates weights and biases, but settings such as learning rate and network depth are selected separately. section 11.2.5 section 11.6.1
Common choices include:
- Architecture: depth, width, activation functions, and connectivity. section 11.6.1
- Training: learning rate, batch size, and regularization strength. section 11.6.1
- Tree models: number of trees or decision-tree depth. section 2.4.1
The learning rate η sets the size of each gradient-descent adjustment; a value that is too large can overshoot, while one that is too small slows training. section 11.2.5
Selection is empirical: train candidate configurations and compare them on validation data, then evaluate the selected configuration on reserved test data. Bayesian optimization can use results from earlier evaluations to choose promising settings, rather than testing every combination. section 11.6.1 section 2.4.1

























