A validation set is held-out data used during model development to assess or tune performance. It is separated from the data shown to the learning algorithm, but remains part of development data; final testing should use data unavailable during modeling. section 8.2.1
When planning an evaluation dataset:
- Set its size in light of variation in key dataset properties. section 6.1.1
- Check that it represents the target domain; an unrepresentative validation set can underspecify what the model should represent. section 6.1.1
- Keep final test data unavailable during modeling: repeated development decisions can optimize against it and produce misleading results. section 8.2.1
