When little data is available, one train/validation split may produce an unstable score because its samples are not representative. Iterated K-fold validation reduces that dependence by repeating shuffled K-fold evaluations and combining the resulting measurements. section 5.2.1 section 6.2.2
From one split to repeated folds
A single shuffled split can place unusually easy or difficult samples in validation. K-fold validation instead divides the data into K equal partitions, trains on K−1 partitions, evaluates on the remaining partition, and averages the K scores. Iterating that process with further shuffling provides a more reliable estimate when the dataset is small. section 5.2.1
Use the protocol with these practical safeguards:
- Choose it when limited data makes hold-out validation unreliable or when split-to-split performance varies substantially. section 5.2.1 section 6.2.2
- Keep validation samples separate from training during each evaluation, and preserve a never-before-seen test set for final assessment. section 5.2.1
- Treat the resulting validation estimate as guidance for model development, not as a substitute for an untouched test-set measurement. section 5.2.1 section 6.2.5
