A performance metric estimates how well a model generalizes and helps guide modeling choices. Its value depends on the evaluation protocol and the metric being measured. section 6.2.2
Choose an evaluation protocol to suit the amount of available data:
- Use a holdout validation set when data is plentiful. section 6.2.2
- Use K-fold cross-validation when a small holdout would make scores unreliable; average the scores across folds. section 4.3.4
- Consider iterated K-fold validation when little data calls for especially accurate evaluation. section 6.2.2
Housing-price validation
Four folds produced validation scores from 0.232 to 0.349, averaging 0.296. That average corresponds to an error of about $29,600, making the metric interpretable against prices of $60,000 to $500,000. Tracking mean absolute error (MAE) across epochs also showed little improvement after roughly 120–140 epochs. section 4.3.4
A strong score on a benchmark does not necessarily establish broad capability: benchmarks may reward memorized answers, and performance on specific tasks can conceal limits in generalization. section 19.2
Listing 4.28 K-fold validation
k = 4 num_val_samples = len(x_train) // k num_epochs = 50 all_scores = [] for i in range(k): print(f"Processing fold #{i + 1}") fold_x_val = x_train[i * num_val_samples : (i + 1) * num_val_samples] #1 fold_y_val = y_train[i * num_val_samples : (i + 1) * num_val_samples] fold_x_train = np.concatenate( #2 [x_train[: i * num_val_samples], x_train[(i + 1) * num_val_samples :]], #2 axis=0, #2 ) #2 fold_y_train = np.concatenate( [y_train[: i * num_val_samples], y_train[(i + 1) * num_val_samples :]], axis=0, ) model = get_model() #3 model.fit( #4 fold_x_train, #4 fold_y_train, #4 epochs=num_epochs, #4 batch_size=16, #4 verbose=0, #4 ) #4 scores = model.evaluate(fold_x_val, fold_y_val, verbose=0) #5 val_loss, val_mae = scores all_scores.append(val_mae)
Inspect the code for splitting folds, training a model on each split, and collecting validation scores. section 4.3.4
