A model’s first configuration is rarely its best. Hyperparameter tuning replaces repeated manual guesses with a systematic search over choices such as layer size and optimizer. Its goal is better validation performance, not better performance on data used to train the model. section 18.1.1
Search a small classifier
For a Keras classifier, define a model-building function that chooses the number of Dense units from 16 to 64 and the optimizer from RMSprop or Adam. KerasTuner can then try configurations and rank them by validation accuracy. section 18.1.1
Listing 18.1 A KerasTuner model-building function
import keras from keras import layers def build_model(hp): units = hp.Int(name="units", min_value=16, max_value=64, step=16) #1 model = keras.Sequential( [ layers.Dense(units, activation="relu"), layers.Dense(10, activation="softmax"), ] ) optimizer = hp.Choice(name="optimizer", values=["rmsprop", "adam"])#2 model.compile( optimizer=optimizer, loss="sparse_categorical_crossentropy", metrics=["accuracy"], ) return model #3
The model-building function makes two decisions searchable while leaving the rest of the classifier fixed. section 18.1.1
A practical tuning loop:
- Choose a configuration, build and train its model, and measure validation performance. Use the results to choose what to try next; repeat within a trial budget. section 18.1.1
- Set an explicit objective, such as validation accuracy. A BayesianOptimization tuner can limit trials and run each configuration more than once. section 18.1.1
- Retrieve promising configurations, determine a suitable training duration, retrain on the full training set, and evaluate on the test set afterward. section 18.1.1
(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data() x_train = x_train.reshape((-1, 28 * 28)).astype("float32") / 255 x_test = x_test.reshape((-1, 28 * 28)).astype("float32") / 255 x_train_full = x_train[:] #1 y_train_full = y_train[:] #1 num_val_samples = 10000 #2 x_train, x_val = x_train[:-num_val_samples], x_train[-num_val_samples:] #2 y_train, y_val = y_train[:-num_val_samples], y_train[-num_val_samples:] #2 callbacks = [ keras.callbacks.EarlyStopping(monitor="val_loss", patience=5), #3 ] tuner.search( #4 x_train, y_train, batch_size=128, epochs=100, validation_data=(x_val, y_val), #3 callbacks=callbacks, verbose=2, )
A validation split and EarlyStopping callback turn each trial into a bounded training run with validation feedback. section 18.1.1
Searching is expensive: unlike weights, configuration choices are often discrete, so they cannot usually be updated by gradient descent. Keep the test set for final evaluation, and beware of overfitting the validation set through repeated tuning. Practice on real problems makes these trade-offs clearer. section 18.1.1 section 18.4
