An architecture prior is a structural assumption built into a model. It favors some kinds of solutions over others. That matters because a network may be able to represent a useful solution without being likely to find it through gradient descent. Choosing an architecture suited to the data helps the model search for patterns that can generalize. section 13.2.3 section 5.3.2
When a dense network is the wrong fit
In a timeseries prediction problem, a densely connected model cannot beat a trivial baseline, while a more appropriate recurrent architecture generalizes well. The lesson is not simply to make the dense model larger: reconsider what the architecture assumes about the task. section 5.3.2
Listing 13.11 Training and evaluating a densely connected model
ncol_input_data <- length(input_data_colnames)
inputs <- keras_input(shape = c(sequence_length, ncol_input_data))
outputs <- inputs |>
layer_reshape(-1) |> #1
layer_dense(16, activation="relu") |>
layer_dense(1)
model <- keras_model(inputs, outputs)
callbacks = list(
callback_model_checkpoint("jena_dense.keras", save_best_only = TRUE) #2
)
model |> compile(
optimizer = "adam",
loss = "mse",
metrics = "mae"
)
history <- model |> fit(
train_dataset,
epochs = 10,
validation_data = val_dataset,
callbacks = callbacks
)
model <- load_model("jena_dense.keras") #3
sprintf("Test MAE: %.2f", evaluate(model, test_dataset)["mae"])
[1] "Test MAE: 2.67"
This R/Keras forecasting baseline reshapes a sequence into a flat input, then applies dense layers. Its reported test mean absolute error is 2.67. The code makes the architecture choice concrete before comparing it with alternatives. section 13.2.3
When choosing an architecture:
- Start with a small model and ask whether it beats a simple baseline. section 6.2.3
- Consider the data modality and relevant prior work when choosing among dense networks, convnets, recurrent networks, and Transformers. section 5.3.2 section 6.2.3
- If reasonable architectures still fail to beat the baseline, do not assume architecture alone is the problem; the task or available data may not support generalization. section 6.2.3 section 5.3.2
