liveBook · Manning

Evaluation dataset

As explained in Build a DeepSeek Model (From Scratch)

A validation set is a held-out subset of data used to measure how well a model generalizes during development.

An evaluation dataset is held-out data reserved for measuring how well a model generalizes to unseen examples during development. A validation set is created by splitting the available corpus before training, rather than using the same examples used to update model weights. section 6.1.3 section 6.3.3

A practical validation-set pipeline typically:

  • Evaluates periodically in evaluation mode with gradient calculation disabled, averaging loss across validation batches. section 6.3.3

TinyStories validation split

A preparation script loads the TinyStories training corpus, assigns 95% of examples to training and 5% to validation, encodes both partitions with the GPT-2 tokenizer, and writes train.bin and val.bin as memory-mapped token files. Training later samples val.bin to estimate generalization.

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains splitting, tokenization, binary storage, periodic loss evaluation, and checkpointing, without detailing statistical confidence or cross-validation.
Examples
Uses a five-percent TinyStories split, GPT-2 tokenization, val.bin storage, validation loss, and best-checkpoint selection.
What it assumes
Assumes a corpus and training loop; teaches the split, encoding, storage, and validation-loss workflow from configuration onward.
What you can do after
Operate/configure a validation pipeline that splits, tokenizes, stores, evaluates, and checkpoints a language-model dataset.
How it connects
Connects train-test splitting, tokenization, generalization, model evaluation, loss, and checkpointing.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.