A hyperparameter is a choice you make about a model or its training before fitting it. The number of layers, units per layer, activation function, dropout amount, and optimizer learning rate are examples. Unlike model parameters, hyperparameters are not learned by backpropagation. A sensible first choice may still be suboptimal, so treat it as something to test rather than a fixed rule. section 18.1.1 section 6.2.5
Choose units and an optimizer
A KerasTuner model-building function can let `hp.Int` choose 16, 32, 48, or 64 units for a Dense layer, while `hp.Choice` selects either `rmsprop` or `adam`. Each combination produces a model to fit on training data and assess on validation data; the choices themselves are not adjusted by backpropagation. section 18.1.1
Listing 18.1 A KerasTuner model-building function
import keras from keras import layers def build_model(hp): units = hp.Int(name="units", min_value=16, max_value=64, step=16) #1 model = keras.Sequential( [ layers.Dense(units, activation="relu"), layers.Dense(10, activation="softmax"), ] ) optimizer = hp.Choice(name="optimizer", values=["rmsprop", "adam"])#2 model.compile( optimizer=optimizer, loss="sparse_categorical_crossentropy", metrics=["accuracy"], ) return model #3
The model-building function shows where candidate hyperparameter values enter the layer and optimizer configuration. section 18.1.1
Tune after a first model shows statistical power and can overfit. Change a configuration, retrain, and compare validation performance; use the validation set, not the test set, to guide changes. KerasTuner can automate the search: its BayesianOptimization example targets `val_accuracy` and limits the search to 20 trials. Repeated tuning can itself overfit the validation process. Once a configuration is satisfactory, retrain on the available training and validation data and evaluate once on the held-out test set. section 6.2.5 section 18.1.1
