Hyperparameter

As explained in Machine Learning Algorithms in Depth

A hyperparameter is a configuration value set outside the ordinary parameter-learning process that controls a model or learning procedure.

A hyperparameter is a configuration value chosen outside ordinary parameter learning that controls a model or learning procedure. Examples include the number of nearest neighbors, the number of K-means clusters, SVM kernel settings, regularization constants, and neural-network learning rates. Hyperparameters shape model behavior and influence whether learned models underfit, overfit, or generalize to unseen data. section 7.4 section 10.4

Tune hyperparameters by searching a multidimensional configuration space for strong validation performance:

• Grid search exhaustively evaluates specified combinations and works well for smaller spaces; logarithmic values can help identify the right order of magnitude. section 7.4
• Random search samples values from prior distributions, can find good configurations faster, and keeps its iteration budget independent of the number of parameters. section 7.4
• Bayesian optimization models generalization performance with a Gaussian process, then selects candidates using expected improvement or an upper confidence bound. section 7.4.1

Learning rate in stochastic gradient descent

The learning rate determines the size of each gradient step. Values that are too high can miss the optimum, while values that are too low prolong training. A practical strategy starts at 1e-3 and reduces the rate with a schedule; common momentum values include 0.5 and 0.9. section 10.4

Figure 7.12 Bayesian optimization algorithm

Use the algorithm figure to trace Bayesian optimization as it balances exploration and exploitation while incorporating previous objective evaluations. section 7.4.1

How Machine Learning Algorithms in Depth covers it

How deep it goes

Defines configuration choices, derives Bayesian optimization with Gaussian processes, and explains learning-rate dynamics without fully detailing every optimizer.

Examples

Uses nearest-neighbor counts, K-means cluster counts, SVM kernels, regularization constants, learning rates, and momentum values.

What you can do after

Apply it via grid search, random search, or Bayesian optimization to configure SVMs, random forests, and neural-network optimizers.

How it connects

Connects model selection, generalization, grid search, random search, Bayesian optimization, Gaussian processes, stochastic gradient descent, momentum, and regularization.

liveBookmanning.com

Read Machine Learning Algorithms in Depth on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Grokking Machine LearningA hyperparameter is a quantity set before training that controls how a machine-learning model is trained or specified.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIA hyperparameter is a configuration value selected before or during model training rather than learned directly from the training examples.
• 

  liveBookmanning.com
  Advanced Algorithms and Data StructuresA hyperparameter is a configuration value governing an algorithm's behavior that is set independently of the algorithm's learned or computed results.
• 

  liveBookmanning.com
  Natural Language Processing in Action, Second EditionA hyperparameter is a user-specified setting that controls a machine-learning algorithm or processing pipeline rather than being learned from its training data.
• 

  liveBookmanning.com
  Deep Learning with R, Third EditionA hyperparameter is a configuration value selected outside parameter learning that controls a model, training procedure, or optimization process.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksA hyperparameter is a configuration value governing model structure or training that is set outside ordinary parameter learning.
• 

  liveBookmanning.com
  Data Without LabelsA hyperparameter is a configuration value selected before or during model training that controls an algorithm's behavior or structure.
• 

  liveBookmanning.com
  Inside Deep Learning: Math, Algorithms, ModelsUser-chosen settings such as layer counts or learning rate that control model architecture or training behavior rather than being learned.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionA user-chosen model or training setting, such as layer count or activation choice, that is set before training and tuned externally rather than learned by backpropagation.
• 

  liveBookmanning.com
  Distributed Machine Learning PatternsA configuration value chosen before or during model training that controls the model structure or training procedure rather than being learned directly from the data.
• 

  liveBookmanning.com
  Transformers in ActionA hyperparameter is a configuration value set before or outside model training that controls architecture or optimization behavior.
• 

  liveBookmanning.com
  Learn Generative AI with PyTorchTraining or model settings chosen before fitting, such as layer counts, head counts, embedding size, vocabulary size, block size, and dropout rate.
• 

  liveBookmanning.com
  Experimentation for EngineersA hyperparameter is a setting that controls the behavior or structure of a statistical model rather than representing an observed system input.
• 

  liveBookmanning.com
  Outlier Detection in PythonA hyperparameter is a configuration value set before model training or algorithm execution that controls the model or algorithm's behavior.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningA hyperparameter is a configuration value selected before or outside the fitting of a machine-learning model.
• 

  liveBookmanning.com
  AI-Powered SearchA hyperparameter is a configuration value that controls a model or learning procedure and is set rather than learned from individual examples.
• 

  liveBookmanning.com
  Designing Deep Learning Systems: A guide for software engineersA configuration value selected before or outside model fitting that controls a learning algorithm or training process.
• 

  liveBookmanning.com
  Machine Learning System DesignA configuration value selected outside model parameter learning that controls training or model behavior.

Compare how 19 books explain hyperparameter →

Related concepts

• parametercontrasts with · A model parameter is a numerical value, such as a weight or bias, that is learned or set to determine how a model computes outputs.
• regularization parameterincludes · A regularization parameter is a model-fitting setting that controls the influence of a penalty or constraint relative to fit to the data.
• temperatureincludes · A temperature parameter adjusts the concentration of a probability distribution, affecting sampling randomness or the likelihood of accepting less favorable transitions.
• learning rateincludes · A learning rate is a step-size parameter that controls the magnitude of updates made by an iterative learning or optimization process.
• grid searchused by · Grid search is a search method that evaluates configurations formed from specified candidate values, typically every combination in their Cartesian product.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Algorithms in Depth. The section links point to the parts of the book it draws on.