Hyperparameter tuning
Hyperparameter tuning is the process of selecting or adjusting configurable model or procedure settings and evaluating them against a chosen objective.
Designing Deep Learning Systems: A guide for software engineers
Hyperparameter optimization in which software algorithms select values and manage trials with minimal human intervention.
- How it is used
- Automates selection of training settings to improve model results within a trial budget.
- How deep it goes
- Explains trial workflows, search categories, service architecture, and library patterns while deferring algorithm mathematics.
- Examples
- Demonstrates classifier selection with Hyperopt, regression tuning with Optuna, and ConvNet tuning with Ray Tune.
- What you can do after
- Apply it via Hyperopt, Optuna, or Ray Tune by defining an objective, search space, algorithm, and trial limit.
Evolutionary Deep Learning: Genetic algorithms and neural networks
Hyperparameter optimization, or hyperparameter tuning, is the process of selecting configuration values that control a machine-learning model or training procedure to improve a chosen objective.
- How it is used
- Improves deep-learning models by selecting training and model control variables through manual or automated search.
- How deep it goes
- Works through runnable random, grid, PSO, evolutionary-strategy, and differential-evolution searches, including Python configuration generators and fitness evaluation.
- Examples
- Uses a PyTorch network approximating a polynomial function while varying epochs, hidden-layer size, learning rate, and batch size.
- What you can do after
- Apply it via Python and DEAP to compare search methods, record fitness, and select promising deep-learning configurations.
Bayesian Optimization in Action
Hyperparameter tuning is the process of selecting machine-learning configuration values that optimize a specified performance objective.
- How it is used
- Uses model evaluations to choose configurations that improve a specified performance metric under limited trials.
- How deep it goes
- Defines queries and expensive black-box objectives, then develops Bayesian search while stopping short of a complete tuning workflow.
- Examples
- Uses neural-network depth, decision-tree settings, SVM parameters, an accuracy table, and simulated SVM accuracy surfaces.
- What you can do after
- Apply it via Bayesian optimization to prioritize promising model configurations when evaluations are expensive.
Read how Bayesian Optimization in Action explains hyperparameter tuning →
Transformers in Action
Hyperparameter optimization is the systematic search for hyperparameter values that improve a model's training or evaluation performance.
- How it is used
- Organizes searches for pretraining or fine-tuning settings that affect training efficiency and model evaluation.
- How deep it goes
- Works through dataset preparation, search configuration, result selection, and retraining, with separate sweep configuration and tracking examples.
- Examples
- Uses CommitmentBank sequence classification, Ray Tune search settings, and a W&B Bayesian sweep with logged evaluation metrics.
- What you can do after
- Apply it via Ray Tune to select a training configuration, or via W&B sweeps to run and compare candidates.
Read how Transformers in Action explains hyperparameter tuning →
Deep Learning with R, Third Edition
Hyperparameter tuning is the process of selecting model configuration values by evaluating alternative settings during development.
- How it is used
- Replaces repeated manual model-configuration tweaks with a systematic search guided by validation performance.
- How deep it goes
- Works through defining a search space, running trials, selecting configurations, and retraining; does not detail Bayesian optimization internals.
- Examples
- Uses an MNIST classifier to search dense-layer width and optimizer choice with KerasTuner.
- What you can do after
- Apply it via KerasTuner to compare configurations, retrain a selected model, and reserve test data for final evaluation.
Read how Deep Learning with R, Third Edition explains hyperparameter tuning →
Deep Learning with Python, Third Edition
The process of searching over model configuration choices to find settings that improve validation or test performance.
- How it is used
- Uses automated search to improve model configurations beyond initial manual choices.
- How deep it goes
- Works through a KerasTuner search and retraining workflow, but does not derive the search algorithms.
- Examples
- Uses an MNIST classifier with tunable Dense units and optimizer, a validation split, and EarlyStopping.
- What you can do after
- Apply it via KerasTuner to search configurations, select promising settings, retrain models, and evaluate them on test data.
Read how Deep Learning with Python, Third Edition explains hyperparameter tuning →
Causal Inference for Data Science
Hyperparameter tuning is the process of selecting model-training settings by evaluating candidate settings and choosing the best-performing configuration.
- How it is used
- Selects machine-learning settings before fitting nuisance models in cross-fitted double machine learning.
- How deep it goes
- Explains candidate evaluation, fold-specific integration, and DoubleML support while deferring exact R and Python syntax to package documentation.
- Examples
- Uses k-nearest-neighbor k values, linear-regression regularization, two-fold DML, and a right-heart catheterization treatment-effect exercise.
- What you can do after
- Operate/configure DoubleML by supplying candidate hyperparameters and enabling fold-specific tuning for an ATE analysis.
Read how Causal Inference for Data Science explains hyperparameter tuning →
Natural Language Processing in Action, Second Edition
Hyperparameter tuning is the systematic process of selecting hyperparameter values to improve a model or processing pipeline.
- How it is used
- Improves NLP pipelines and neural-network models by selecting configurations that produce better retrieval or predictive performance.
- How deep it goes
- Defines tuning, demonstrates parameter changes, compares logged experiments, and discusses Bayesian automation without prescribing a complete search algorithm.
- Examples
- Uses CountVectorizer 2-grams, a command-line CNN search, learning curves, and logged RNN comparisons.
- What you can do after
- Apply it via recorded experiments and parameter comparisons to improve retrieval, accuracy, training speed, or overfitting behavior.
Read how Natural Language Processing in Action, Second Edition explains hyperparameter tuning →
Machine Learning System Design
Hyperparameter optimization is the process of searching for configuration values that improve a model's performance.
- How it is used
- Guides training-pipeline experiments toward configurations that improve model performance under practical compute constraints.
- How deep it goes
- Explains parameter selection, manual and automated search, experiment tracking, and validation risks without deriving search algorithms.
- Examples
- Uses retail forecasting, grid search, Bayesian optimization, random search, MLflow tracking, and hierarchical configuration files.
- What you can do after
- Apply it via random search, Bayesian optimization, grid search, or controlled manual experiments while tracking results and protecting evaluation data.
Read how Machine Learning System Design explains hyperparameter tuning →
Advanced Algorithms and Data Structures
Parameter fine-tuning is the systematic refinement of tested parameter values around the most promising settings.
- How it is used
- Uses systematic parameter search to improve genetic-algorithm results.
- How deep it goes
- Explains a coarse-to-fine protocol and combination costs, but does not develop a formal optimization method.
- Examples
- Uses mutation-chance ranges and a TSP table comparing crossover and swap-operator chances.
- What you can do after
- Apply coarse-to-fine search to select promising values for several genetic-algorithm parameters.
Read how Advanced Algorithms and Data Structures explains hyperparameter tuning →
Data Analysis with LLMs
Model tuning is the adjustment of a model's generation settings to improve its behavior on a task.
- How it is used
- Adjusts generation settings to constrain review-label outputs while improving classification quality and reducing unused token generation.
- How deep it goes
- Demonstrates token restrictions and a one-token output limit, but notes these are only a subset of available tuning options.
- Examples
- Compares ten review labels and reports seven correct results after limiting output to one token, up from six.
- What you can do after
- Apply output constraints through the OpenAI Python client to generate single-token class labels and limit response length.
Read how Data Analysis with LLMs explains hyperparameter tuning →
Machine Learning for Tabular Data: XGBoost, Deep Learning, and AI
Hyperparameter optimization, or tuning, is the process of searching for hyperparameter settings that improve a model's evaluation objective.
- How it is used
- Improves model evaluation results by systematically testing hyperparameter settings within a defined search space.
- How deep it goes
- Explains search-space design, cross-validation, algorithm choice, and a 60-trial optimization, without deriving search algorithms mathematically.
- Examples
- Uses a Tokyo Airbnb XGBoost regressor, Optuna trials, MAE, booster choices, objectives, and parameter-importance visualizations.
- What you can do after
- Apply it via Optuna to evaluate candidate XGBoost configurations and retrieve suitable hyperparameters for a regression objective.
Distributed Machine Learning Patterns
Hyperparameter tuning is the process of selecting hyperparameter values by evaluating models trained with different configurations.
- How it is used
- Optimizes TensorFlow model configurations before selecting a final model for evaluation.
- How deep it goes
- Explains search spaces, Hyperband, validation objectives, early stopping, and final retraining without detailing tuner internals.
- Examples
- Uses an MNIST dense neural network with tunable layer units and Adam learning rates.
- What you can do after
- Apply it via Keras Tuner to search configurations, retrieve the best hyperparameters, retrain, and evaluate a model.
Read how Distributed Machine Learning Patterns explains hyperparameter tuning →
Optimization Algorithms: AI techniques for design, planning, and control problems
Parameter tuning is the process of selecting suitable values for an algorithm's configurable parameters before or during execution.
- How it is used
- Selects or updates algorithm parameters to improve search performance for problem-specific optimization tasks.
- How deep it goes
- Explains timing, update modes, parameter trade-offs, and sensitivity analysis through tabu search and particle swarm optimization.
- Examples
- Uses tabu tenure, swarm size, and neighborhood size to illustrate tuning trade-offs.
- What you can do after
- Apply it via experimentation, sensitivity analysis, or adaptive updates to choose problem-specific algorithm parameters.
Inside Deep Learning: Math, Algorithms, Models
The process of searching over user-chosen model settings to find values that improve an objective score.
- How it is used
- Uses bounded trials to improve model accuracy or error through automated sequential search.
- How deep it goes
- Defines the objective-and-trial workflow but defers the technical details of Optuna’s Bayesian model.
- Examples
- Demonstrates Optuna on an absolute-value toy function and warm-started searches over linear-model penalties.
- What you can do after
- Apply it via Optuna by defining a scalar objective, bounding suggestions, running trials, and reading the best parameters.
Read how Inside Deep Learning: Math, Algorithms, Models explains hyperparameter tuning →
LLMs in Production
Hyperparameter tuning is the process of selecting values for training or model-configuration choices that are not learned as model parameters.
- How it is used
- Guides choices among training settings and generation controls when improving model performance or response behavior.
- How deep it goes
- Defines temperature through logits and softmax, but treats broad LLM tuning briefly and defers deeper optimization to data and model changes.
- Examples
- Uses Optuna, the prompt “I am a,” temperature settings, logits, softmax probabilities, and beam search as teaching examples.
- What you can do after
- Apply it via Optuna for training searches or configure temperature and beam settings through a compatible LLM API endpoint.
Read how LLMs in Production explains hyperparameter tuning →
Machine Learning Platform Engineering
Hyperparameter optimization is the process of selecting hyperparameter values by evaluating configurations within a defined search space against an objective such as a loss function.
- How it is used
- Deploys automated configuration search during model training to improve target metrics while reducing manual steps.
- How deep it goes
- Defines search spaces, objectives, automation, and modularity but defers specific search algorithms and implementation details.
- Examples
- Uses a defined search space, a loss function, tracked checkpoints, and a retraining pipeline with versioned components.
- What you can do after
- Operate automated searches reproducibly by supplying configurations, data, code, a search space, and an objective.
Read how Machine Learning Platform Engineering explains hyperparameter tuning →
Financial AI in Practice
Hyperparameter tuning is the process of selecting configuration values for a model’s learning procedure to improve its performance.
- How it is used
- Uses search to select model-learning settings against a task-specific validation metric, such as fraud recall.
- How deep it goes
- Works through randomized search, fold-level resampling, refitting, and test evaluation; mentions grid and Bayesian search without implementing them.
- Examples
- Uses a fraud pipeline tuned for recall, fixed XGBoost settings in credit scoring, and conservative random forest settings for market prediction.
- What you can do after
- Apply scikit-learn RandomizedSearchCV to select and refit a fraud-detection pipeline before evaluating it on unseen test data.
Read how Financial AI in Practice explains hyperparameter tuning →
Related concepts
- particle swarm optimizationused by · Particle swarm optimization is a population-based optimization method in which candidate solutions adjust their positions using individual and collective search information.
- hyperparameteruses · A hyperparameter is a setting chosen outside ordinary parameter learning that controls a model, algorithm, or learning procedure.
- Optunauses · Optuna is an open-source Python framework for automated hyperparameter optimization through repeated trials that explore parameter configurations.
- KerasTunerused by · KerasTuner is a library for defining, running, and evaluating automated searches for effective hyperparameter configurations of Keras models.
- objective functionuses · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.