liveBook · Manning

Evaluation dataset

As explained in LLMs in Production

A validation dataset is a held-out collection of examples used to assess and tune a model during development.

A validation dataset is held-out data used after training to check whether a model generalizes to unseen examples. Its results can guide parameter fine-tuning and architecture adjustments during development. section 5.2.1

A practical development loop uses the validation set to:

  • Evaluate a newly trained language model on data it did not train on. section 5.2.1
  • Compare fine-tuned parameters or architecture changes and repeat the process as needed. section 5.2.1
Figure 5.2 A simplified version of all the steps necessary to train a language model (large or otherwise) from scratch. You must have data, then define all of the model behavior, and only then proceed to train.
figure

The training pipeline places data and model definition before training, followed by validation of the resulting model. section 5.2.1

How LLMs in Production covers it

How deep it goes
Explains evaluation and iterative tuning after training without detailing validation metrics or dataset-construction procedures.
Examples
Uses a language model trained from scratch and repeated parameter or architecture adjustments as examples.
What it assumes
Assumes training data and a trained model, while treating tokenization and embeddings as prior concepts.
What you can do after
Evaluate a trained model on unseen data and use results to guide fine-tuning or architecture changes.
How it connects
Connects model training with generalization, fine-tuning, and architecture adjustment.
Cover of LLMs in Production
liveBookmanning.com
Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

Compare how 27 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.