A baseline is a simple reference score that a new model should beat. Set one before training, using the same validation metric you will use for the model. Otherwise, a score that looks promising may not show any useful learning. section 5.2.2
Predict tomorrow’s temperature from today’s
For a temperature forecast 24 hours ahead, predict that the temperature will be the same as it is now. Measure the absolute difference between each prediction and its target, then average those differences: mean(abs(preds - targets)). This no-learning rule produces a validation mean absolute error (MAE) of 2.43°C and a test MAE of 2.62°C. A more complicated model needs to improve on this reference to justify its complexity. section 13.2.2 section 13.5
Choose a baseline that fits the task:
- For balanced digit or sentiment classification, compare validation accuracy with random guessing: 0.1 for ten MNIST digits or 0.5 for binary IMDb sentiment. section 5.2.2
- For imbalanced classification, check the majority-class rule. If 90% of examples are class A, always predicting A already achieves 90% accuracy. section 5.2.2
- For a forecast with an obvious persistence rule, evaluate that rule before training a network. section 13.2.2
Listing 13.11 Training and evaluating a densely connected model
ncol_input_data <- length(input_data_colnames)
inputs <- keras_input(shape = c(sequence_length, ncol_input_data))
outputs <- inputs |>
layer_reshape(-1) |> #1
layer_dense(16, activation="relu") |>
layer_dense(1)
model <- keras_model(inputs, outputs)
callbacks = list(
callback_model_checkpoint("jena_dense.keras", save_best_only = TRUE) #2
)
model |> compile(
optimizer = "adam",
loss = "mse",
metrics = "mae"
)
history <- model |> fit(
train_dataset,
epochs = 10,
validation_data = val_dataset,
callbacks = callbacks
)
model <- load_model("jena_dense.keras") #3
sprintf("Test MAE: %.2f", evaluate(model, test_dataset)["mae"])
[1] "Test MAE: 2.67"
This small Keras dense network is the next comparison for the temperature task. Its reported test MAE is 2.67°C, versus 2.62°C for the no-learning rule: training a model does not guarantee an improvement. section 13.2.3 section 13.2.2
Beating the baseline is an early milestone, not the end of model development: it indicates that the model uses input information to make predictions that generalize. If several reasonable models cannot beat it, revisit the features, architecture, training setup, or whether machine learning suits the problem. Once a small model clears that threshold, test whether added capacity and later tuning improve validation performance. section 5.2.2 section 6.2.3 section 6.4
