A hyperparameter is a user-chosen setting that controls a model’s architecture or training behavior rather than being learned from gradients. Examples include learning rate, momentum, hidden-layer count, and neurons per hidden layer. section 5.4 section 2.2.1
Hyperparameters are optimized by training models under different settings and comparing a validation result. Grid search becomes expensive as variables multiply, so Optuna suggests values within a computational budget. section 5.4 section 5.4.2
A practical PyTorch tuning workflow is:
- Create new training and validation splits from the original training data, keeping the true validation set unused until final evaluation. section 5.4.2
- Ask Optuna for candidate values, such as neurons per layer, hidden-layer count, and learning rate. section 5.4.2
- Build the network from those values, train it, and return the validation result as the objective value. section 5.4.2
- Run multiple trials and inspect the best parameters and optimization history. section 5.4.2
Dynamic fully connected network
A PyTorch model can construct a variable number of hidden layers by appending Linear and activation modules inside a loop, then passing the resulting list to nn.Sequential. Optuna can supply values such as neurons_per_layer between 16 and 256 and hidden_layers between 1 and 6. section 5.4.2
Figure 5.17 The four steps of defining an objective function that Optuna can use to optimize our neural network training. The two most important steps shown as code are getting the hyperparameters and computing the result!
Use the four-step objective-function diagram to connect parameter suggestions, model construction, training, and validation scoring. section 5.4.2
