liveBook · Manning

Dataset split

As explained in LLM Customization and Fine-Tuning

A subset of a dataset assigned a role such as training, validation, or testing.

A dataset split assigns examples distinct roles: train on the training set, use validation during iteration, and reserve the test set for final evaluation. Keeping test data out of training and model selection makes its results reportable. section 5.1.1

IT support fine-tuning data

A reproducible setup uses 450 training, 50 validation, and 50 test examples. The test set stays untouched until evaluation. section 5.1.1

python scripts/build_it_support_dataset.py \
    --out data/it_support \
    --seed 42 \
    --train 450 --valid 50 --test 50

Generate the example splits with a fixed seed and explicit counts. section 5.1.1

How LLM Customization and Fine-Tuning covers it

How deep it goes
Specifies split generation and test-set isolation, but does not compare alternative splitting strategies.
Examples
Uses an IT support dataset with 450 training, 50 validation, and 50 test examples.
What you can do after
Evaluate fine-tuning results with a test set kept outside training and model selection.
How it connects
Connects dataset splits to fine-tuning, validation, and model selection.

Also explained in

Compare how 24 books explain dataset split →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.