An evaluation dataset is held-out data reserved for measuring how well a model generalizes to unseen examples during development. A validation set is created by splitting the available corpus before training, rather than using the same examples used to update model weights. section 6.1.3 section 6.3.3
A practical validation-set pipeline typically:
- Evaluates periodically in evaluation mode with gradient calculation disabled, averaging loss across validation batches. section 6.3.3
TinyStories validation split
A preparation script loads the TinyStories training corpus, assigns 95% of examples to training and 5% to validation, encodes both partitions with the GPT-2 tokenizer, and writes train.bin and val.bin as memory-mapped token files. Training later samples val.bin to estimate generalization.








































