A hyperparameter is a configuration value selected for model structure or training rather than learned directly from data. It remains fixed during a training run and can directly affect model performance. Typical choices include neural-network layer counts, hidden units, learning rate, and tree maximum depth. section 5.4 section 8.1.1
Changing hyperparameters commonly requires a new training run. In an end-to-end workflow, rerunning unchanged data ingestion just to test a new configuration is time-consuming; step memoization addresses this by skipping redundant work while experiments vary model architecture or configuration. section 5.4
A practical tuning flow is:
- Expose candidate values in the model-building function, such as 32–512 units in steps of 32 and a choice of learning rates. section 8.1.1
- Run a tuner against validation accuracy; Hyperband uses adaptive resource allocation and early stopping to find a high-performing model faster. section 8.1.1
- Retrieve the best configuration, build its model, train it further, and evaluate it on test data. section 8.1.1
Tuning an MNIST classifier
A Keras model exposes the first dense layer's unit count and Adam learning rate as hyperparameters. Hyperband searches these choices with validation data and an EarlyStopping callback, then the selected configuration is used to train and evaluate the final classifier. section 8.1.1
Listing 8.15 The model building function using TensorFlow and Keras Tuner
def model_builder(hp):
model = keras.Sequential()
model.add(keras.layers.Flatten(input_shape=(28, 28)))
hp_units = hp.Int('units', min_value=32, max_value=512, step=32)
model.add(keras.layers.Dense(units=hp_units, activation='relu'))
model.add(keras.layers.Dense(10))
hp_learning_rate = hp.Choice('learning_rate', values=[1e-2, 1e-3, 1e-4])
model.compile(optimizer=keras.optimizers.Adam(learning_rate=hp_learning_rate),
loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),
metrics=['accuracy'])
return model
The model-builder listing shows how named search-space values are inserted into a dense layer and Adam optimizer. section 8.1.1
