Model size reflects a model’s capacity, shaped by its number of layers and the size of those layers. Too little capacity can lead to underfitting; too much can make a model memorize training data and overfit. section 5.4.4
Find an appropriate size by comparing architectures:
- Start with relatively few layers and parameters, then enlarge layers or add layers until validation loss shows diminishing returns. Compare on validation data, not test data. section 5.4.4
- Watch both training and validation loss: greater capacity can drive training loss down quickly while widening the gap between training and validation performance. section 5.4.4
Listing 5.11 Version of the model with lower capacity
model <- keras_model_sequential() |>
layer_dense(4, activation = "relu") |>
layer_dense(4, activation = "relu") |>
layer_dense(1, activation = "sigmoid")
model |> compile(
optimizer = "rmsprop",
loss = "binary_crossentropy",
metrics = "accuracy"
)
history_smaller_model <- model |> fit(
train_data, train_labels,
epochs = 20, batch_size = 512, validation_split = 0.4
)plot(
NULL,
main = "Original Model vs. Smaller Model on IMDB Review Classification",
xlab = "Epochs",
xlim = c(1, history_original$params$epochs),
ylab = "Validation Loss",
ylim = extendrange(c(history_original$metrics$val_loss,
history_smaller_model$metrics$val_loss)),
panel.first = abline(v = 1:history_original$params$epochs,
lty = "dotted", col = "lightgrey")
)
lines(history_original $metrics$val_loss, lty = 2)
lines(history_smaller_model$metrics$val_loss, lty = 1)
legend("topleft", lty = 2:1,
legend = c("Validation loss of original model",
"Validation loss of smaller model"))
Compare the smaller IMDb review model, with four units in each hidden layer, against the much larger version with 512; the accompanying validation-loss comparisons show how their behavior differs. section 5.4.4
