Expressive power, also called representational power, is a model’s ability to represent patterns in data. Consider classifying handwritten digits from their pixels. A model must be expressive enough to fit the training examples, but fitting alone does not show that it has learned a pattern that works on new digits. section 5.1.2 section 5.3.3
Try a small model, then a larger one
On MNIST, a single dense softmax layer behaves like logistic regression: its validation loss stalls around 0.26, even after continued training. Replace it with two intermediate dense layers of 128 units each, and the model fits quickly; its validation loss begins to rise after about eight epochs. The first model needs more capacity. The second has enough capacity to reveal when overfitting starts. section 5.3.3
model <- keras_model_sequential() |>
layer_dense(128, activation = "relu") |>
layer_dense(128, activation = "relu") |>
layer_dense(10, activation = "softmax")
model |> compile(
optimizer = "rmsprop",
loss = "sparse_categorical_crossentropy",
metrics = "accuracy"
)
history_large_model <- model |> fit(
train_images, train_labels,
epochs = 20,
batch_size = 128,
validation_split = 0.2
)plot(history_large_model$metrics$val_loss, type = 'o',
main = "Validation Loss for a Model with Appropriate Capacity",
xlab = "Epochs", ylab = "Validation Loss")
The R/Keras listing builds and trains the two-layer MNIST model and plots validation loss, making the effect of added capacity visible. section 5.3.3
More power is not automatically better. A model can fit MNIST images paired with randomly shuffled labels: training loss falls, but validation loss does not improve. Enough parameters let it memorize examples without learning a relationship that generalizes. section 5.1.2
Listing 5.4 Fitting an MNIST model with randomly shuffled labels
.[.[train_images, train_labels], .] <- dataset_mnist()
train_images <- array_reshape(train_images / 255,
c(60000, 28 * 28))
random_train_labels <- sample(train_labels) #1
model <- keras_model_sequential() |>
layer_dense(512, activation = "relu") |>
layer_dense(10, activation = "softmax")
model |> compile(optimizer = "rmsprop",
loss = "sparse_categorical_crossentropy",
metrics = "accuracy")
history <- model |> fit(
train_images, random_train_labels,
epochs = 100, batch_size = 128,
validation_split = 0.2
)
The shuffled-label experiment is a practical check on the difference between the ability to fit data and the ability to generalize. section 5.1.2
Use training and validation behavior to adjust expressive power:
- If a model fits but cannot be made to overfit, try more layers, wider layers, or layer types better suited to the problem. section 5.3.3
- If it overfits almost immediately, compare smaller architectures on validation loss; consider dropout or weight regularization. Keep the test set separate from these choices. section 5.4.4 section 20.1.5
Expressive power also depends on architecture, not just size. The contrast between shallow word2vec and Transformers illustrates this: stacked attention and feedforward layers support much more complex transformations in token-vector space than the toy model’s simple vector relationships. section 15.5
