K-fold cross-validation estimates model performance when a small validation set would be unreliable. It divides the data into K partitions, trains K models, and evaluates each model on a different held-out partition. Averaging the K validation scores reduces dependence on one arbitrary split. section 4.3.4 section 5.2.1
A four-fold validation loop
Set k <- 4 and assign every training example to a fold. For each fold, use its rows as validation data, use all other rows for training, create a fresh model, fit it, and record its mean absolute error. Scores such as 0.336, 0.290, 0.266, and 0.268 produce an average of 0.290, which is more reliable than any single score. section 4.3.4 section 5.2.1
Use K-fold cross-validation as an evaluation workflow:
- Choose K, typically 4 or 5 when little data is available. section 4.3.4
- Create identical models for each fold so every validation score comes from an independently trained model. section 4.3.4
- Average the fold scores, then inspect the per-epoch average when choosing a training duration. section 4.3.4
- After selecting the training setup, fit a final model on all available training data and evaluate it once on untouched test data. section 5.2.1
- Prefer K-fold validation over hold-out validation when too few samples make a single split statistically unrepresentative. section 6.2.2
Figure 4.8 3-fold cross-validation
Use the three-fold diagram to see how each partition takes a turn as validation data while the remaining partitions provide training data. section 5.2.1
Listing 4.25 K-fold validation
k <- 4
fold_id <- sample(rep(1:k, length.out = nrow(train_data)))
num_epochs <- 50
all_scores <- numeric(k)
for (i in 1:k) {
cat(sprintf("Processing fold #%i\n", i))
fold_val_indices <- which(fold_id == i)
fold_x_val <- x_train[fold_val_indices, ] #1
fold_y_val <- y_train[fold_val_indices]
fold_x_train <- x_train[-fold_val_indices, ] #2
fold_y_train <- y_train[-fold_val_indices] #2
model <- get_model() #3
model |> fit( #4
fold_x_train, fold_y_train,
epochs = num_epochs, batch_size = 16, verbose = 0
)
results <- model |> evaluate(fold_x_val, fold_y_val, verbose = 0) #5
all_scores[i] <- results$mean_absolute_error
}
Processing fold #1
Processing fold #2
Processing fold #3
Processing fold #4
Use the runnable listing to follow fold assignment, complementary training indices, fresh model creation, fitting, evaluation, and score collection. section 4.3.4
