Iterated K-fold validation with shuffling estimates generalization by repeating K-fold validation over multiple randomly shuffled partitions. It is especially useful when little data makes a single validation split unreliable or performance varies substantially with the split. section 5.2.1 section 6.2.2
A typical evaluation run follows this pattern:
- Shuffle the data and assign samples to K approximately equal folds. section 5.2.1
- For each fold, train a fresh model on the other K − 1 folds and evaluate it on the held-out fold. section 5.2.1
- Average the K validation scores, then repeat the whole process with new shuffles or partitions and combine the resulting estimates. section 5.2.1 section 5.5
- Keep a completely separate test set for final evaluation, rather than tuning decisions against it. section 5.2.1 section 5.5
Three-fold cross-validation as the baseline
With k <- 3, fold assignments divide the data into three folds. The loop trains and evaluates three fresh models, stores three validation scores, and averages them before fitting a final model on all available data. Iterated validation repeats this complete shuffled procedure to obtain a more stable estimate. section 5.2.1
Listing 5.6 K-fold cross-validation (labels omitted for simplicity)
k <- 3
fold_id <- sample(rep(1:k, length.out = nrow(data)))
validation_scores <- numeric(k)
for (fold in seq_len(k)) {
validation_idx <- which(fold_id == fold) #1
validation_data <- data[validation_idx, ] #1
training_data <- data[-validation_idx, ] #2
model <- get_model() #3
fit(model, training_data, ...)
validation_score <- evaluate(model, validation_data, ...)
validation_scores[[fold]] <- validation_score
}
validation_score <- mean(validation_scores) #4
model <- get_model() #5
fit(model, data, ...)
test_score <- evaluate(model, test_data, ...)
Inspect the R-style K-fold implementation showing fold assignment, held-out indices, fresh models, score averaging, and final fitting. section 5.2.1
