A hyperparameter is a choice you make about a model or its training before its weights are learned. The number of layers and units per layer are hyperparameters; the weights learned during training are model parameters. The optimizer and its learning rate are choices you can tune, too. section 5.2.1 section 6.2.5 section 18.1.1
Choose units and an optimizer
For a model trained on MNIST, a KerasTuner model-building function can let `hp$Int()` choose 16, 32, 48, or 64 units for a dense layer. `hp$Choice()` can choose between `rmsprop` and `adam`. Each choice produces a model to train and compare—not a weight learned by that model. section 18.1.1
Listing 18.1 KerasTuner model-building function
build_model <- function(hp, num_classes = 10) {
units <- hp$Int( #1
name = "units", #1
min_value = 16L, #1
max_value = 64L, #1
step = 16L) #1
model <- keras_model_sequential() |> #1
layer_dense(units, activation = "relu") |>
layer_dense(num_classes, activation = "softmax")
optimizer <- hp$Choice(name = "optimizer", #2
values = c("rmsprop", "adam"))
model |> compile(optimizer = optimizer,
loss = "sparse_categorical_crossentropy",
metrics = "accuracy")
model #3
}
The model-building function shows where two searchable hyperparameters enter an R/Keras model: dense-layer units and optimizer choice. section 18.1.1
A practical tuning loop separates model selection from final evaluation:
- Train each candidate on the training set and compare its performance on the validation set. A tuner can choose the next configuration automatically. section 18.1.1
- Once the configuration is satisfactory, train a final model on the available training and validation data, then evaluate it on the untouched test set. section 6.2.5
- Watch for validation-set overfitting: repeated changes based on validation results leak information into your choices, even though the model is never trained directly on validation data. section 5.2.1
Tune after a simple baseline works and the model has enough capacity to overfit. Try architecture changes, regularization, and learning-rate changes to improve generalization. If test performance is much worse than validation performance, reconsider the validation procedure or whether tuning has overspecialized the configuration. section 6.2.5 section 6.4
