A validation set is held-out data for comparing models and selecting the one that best serves the task. Keep it separate from training data, or memorization can make a weak model look capable. section 3.9
Comparing models
To choose between GPT-4o and Llama 3, measure how often each solves coding problems correctly on the same validation set. That task-focused measure may differ from the loss used during training. section 3.9
Use a validation set carefully:
- Exclude examples already present in training data; otherwise, memorization and overfitting can inflate performance. section 3.9
- After selecting a model, evaluate it once on a separate test set as a final sanity check. section 3.10
- Do not reuse the test set after disappointing results; repeated use turns it into another validation set. section 3.10
