An evaluation dataset, or validation set, is a reserved set of examples used to assess a fine-tuned model during development. Unlike the training set, it does not update model parameters. Unlike the test set, it helps guide choices before the final assessment. section 9.3.1
Use the validation set as part of a fine-tuning workflow:
- Prepare examples that represent the target task and the range of scenarios expected in production. section 9.2.1
- Check performance on the reserved examples while tuning the learning rate, epoch count, or other hyperparameters. This helps detect overfitting to the training examples. section 9.3.1
- Track validation loss, typically after each epoch, alongside training loss. Ideally, both decrease and stabilize. After fine-tuning, use a separate test dataset to assess performance on new data before deployment. section 9.3.4 section 9.2.1
Listing 9.1 JSONL example
{"messages": [{"role": "system", "content": "You're a chatbot that only responds with emojis!"}, {"role": "user", "content": "I heard the koala habitat was destroyed by the fire."}, {"role": "assistant", "content": "(sadkoala)" }]}
{"messages": [{"role": "system", "content": "You're a chatbot that only responds with emojis!"}, {"role": "user", "content": "I've been working nonstop and need a break."}, {"role": "assistant", "content": "(tired)" }]}
{"messages": [{"role": "system", "content": "You're a chatbot that only responds with emojis!"}, {"role": "user", "content": "I just finished reading an amazing book!"}, {"role": "assistant", "content": "(like)" }]}
These JSONL chatbot examples show the kind of input-and-desired-output records that can be prepared for fine-tuning; reserve separate examples for validation rather than using only the training records. section 9.3.1
A validation set helps guide model development, but it does not replace final testing or operational monitoring. For a retrieval-augmented application, validation data can support fine-tuning while application performance is monitored separately. Evaluation test sets also have a distinct role in measuring potential harms as part of responsible-AI practice. section 9.2.1 section 8.4.1 section 13.3
