A validation set, or dev set, is data held out from fine-tuning. Evaluate the model on it during training, but do not use it to fit the model. This helps reveal overfitting: strong performance on training examples that does not carry over to unseen inputs. section 5.3.1
To make the check useful:
- Set aside about 10–20% of the data, with a distribution that mirrors the training set. section 5.3.1
- Keep validation examples out of the training set; overlap creates data leakage. section 5.3.1
- Monitor validation accuracy or loss during training. An OpenAI fine-tuning validation file provides reported metrics; custom training code can compute them. section 5.3.1
A stopping signal
If validation loss starts increasing, consider stopping training: the model is likely overfitting. section 5.3.1
