A baseline is a simple prediction rule whose score gives a model something meaningful to beat. Without one, a validation score is hard to interpret. For example, if 90% of samples belong to class A, always predicting A achieves 90% accuracy. A classifier scoring 85% has not yet justified its complexity. section 5.2.2
Tomorrow’s temperature: predict today’s
For a 24-hour temperature forecast, a commonsense rule predicts that the temperature will be the same as it is now. Measured by mean absolute error (MAE), this rule scores 2.44°C on validation data and 2.62°C on test data. A later 1D convolutional model scores about 2.9°C validation MAE—worse than the simple rule. section 13.2.2 section 13.2.4
Listing 13.9 Computing the commonsense baseline MAE
def evaluate_naive_method(dataset): total_abs_err = 0.0 samples_seen = 0 for samples, targets in dataset: preds = samples[:, -1, 1] * std[1] + mean[1] #1 total_abs_err += np.sum(np.abs(preds - targets)) samples_seen += samples.shape[0] return total_abs_err / samples_seen print(f"Validation MAE: {evaluate_naive_method(val_dataset):.2f}") print(f"Test MAE: {evaluate_naive_method(test_dataset):.2f}")
The evaluation loop turns the latest observed temperature into each prediction, totals the absolute errors, and divides by the number of samples. section 13.2.2
Use the baseline as an early decision point:
- Choose the evaluation metric and a trivial rule before developing the model. A random classifier or a simple non-machine-learning technique may be enough. section 5.2.2 section 6.4
- Compare scores on validation data. Beating the baseline is the first sign of generalization power, not the end of model development. section 5.2.2 section 5.3
- If reasonable models cannot beat it, reconsider features, architecture, training configuration, or whether machine learning suits the problem. Try simple models before paying for more expensive ones. section 6.2.3 section 13.5
