Expressive power is a model’s ability to represent the patterns a task requires. A model learns transformations of its inputs into useful representations; in a neural network, successive layers can learn successive representations. More layers are one way to increase expressive power, but the kind of layer matters too. section 1.4 section 1.5 section 5.3.3
When a model cannot overfit
A simple logistic regression trained on MNIST pixels reaches a validation loss of about 0.26 that stalls rather than rising after a peak. Even with more training, it does not clearly overfit. Replacing it with a network containing two intermediate layers of 128 units lets it fit quickly and begin overfitting after eight epochs. That change is evidence that the smaller model lacked capacity—not that overfitting is the final goal. section 5.3.3 section 5.5
Use the training and validation curves to decide what to change:
- If the model cannot clearly overfit, try more units, more layers, or layer types better suited to the problem. section 5.3.3
- If it overfits immediately, more capacity may only add memorization; turn to regularization or reduce capacity. section 5.3.3 section 5.5
- If performance stalls without serious overfitting, increasing capacity is worth testing—but expect possible diminishing returns. section 13.3.5
The same decision applies to sequence models. Stacking two dropout-regularized GRU layers improved a temperature forecast to a test MAE of 2.39 degrees, an 8.8% improvement over the baseline, but the added layer helped only modestly. In a Keras stack, the first GRU must return its full output sequence so the next GRU can process it. section 13.3.5
Listing 13.23 Training and evaluating a dropout-regularized, stacked GRU model
inputs = keras.Input(shape=(sequence_length, raw_data.shape[-1])) x = layers.GRU(32, recurrent_dropout=0.5, return_sequences=True)(inputs) x = layers.GRU(32, recurrent_dropout=0.5)(x) x = layers.Dropout(0.5)(x) outputs = layers.Dense(1)(x) model = keras.Model(inputs, outputs) callbacks = [ keras.callbacks.ModelCheckpoint( "jena_stacked_gru_dropout.keras", save_best_only=True ) ] model.compile(optimizer="adam", loss="mse", metrics=["mae"]) history = model.fit( train_dataset, epochs=50, validation_data=val_dataset, callbacks=callbacks, ) model = keras.models.load_model("jena_stacked_gru_dropout.keras") print(f"Test MAE: {model.evaluate(test_dataset)[1]:.2f}")
The stacked GRU listing shows the capacity change alongside dropout and the intermediate layer’s return_sequences=True setting. section 13.3.5
