Hyperparameter tuning selects the control variables that define a machine-learning model or its training process. In deep learning, these are distinct from learned weights and can include learning rate, epochs, batch size, hidden-layer size, optimizer, and loss function. Because models rarely work well unchanged, choosing these settings is often a central part of improving them. section 1.3 section 1.5.3 section 5.1
Treat tuning as a search problem: generate a configuration, train and evaluate the model, retain its fitness or loss, and compare runs. Manual adjustment can build intuition, but it is slow and error-prone. More importantly, changing a critical option such as model architecture, optimizer, or loss function invalidates earlier assumptions and calls for retuning. section 5.1.2 section 5.7
Common search approaches trade coverage, speed, and behavior differently:
- Random search samples parameter values from defined ranges. It can quickly explore possibilities and reveal promising regions, but large runs can be expensive. section 5.2.1 section 5.3
- Grid search evaluates discrete combinations systematically across a search space. Visualizing results can guide further refinement of selected regions. section 5.3 section 5.7
- Particle swarm optimization searches with a population of particles and is suited to problems with relatively few hyperparameters. section 5.4.1 section 5.7
- Evolutionary strategies evolve a population through blending and mutation; they can converge quickly but may cluster around local optima. section 5.5.1
- Differential evolution forms new candidates from scaled differences among sampled population members, aiming to avoid clustering seen with PSO and evolutionary strategies. section 5.6.1 section 5.7
Random-search loop
A simple Python implementation stores generators for each hyperparameter, draws a child configuration for every run, trains a PyTorch network, measures loss as fitness, records the result, and retains the lowest-loss configuration. One setup samples epochs from 20–400, middle-layer width from 8–64, learning rate from 0.0035–0.35, and batch size from 4–64. section 5.2.1
Evolutionary HPO can map each candidate vector to values in bounded hyperparameter ranges, then train and score the resulting network. DEAP supplies the evolutionary machinery for particle swarm optimization, evolutionary strategies, and differential evolution. For high-dimensional searches, dimensionality reduction such as PCA can help visualize differences between methods, but visualization supports judgment rather than proving a universal winner. section 5.4.2 section 5.7
Compare random sampling with the methodical traversal of a grid over learning rate and hidden-layer size. section 5.3
