An evaluation dataset is held-out data reserved for checking a trained model on inputs and targets it did not use during training. Comparing predictions with known labels reveals accuracy and prediction error. section 4.5.2 section 4.5.3
Use a disciplined split rather than treating every available row as training data:
- Shuffle first when rows contain implicit ordering such as dates, categories, or alphabetical ranges. section 4.5.2
- Allocate separate training and evaluation portions, such as 80/20, and retain features and labels for both portions. section 4.5.2
- Run prediction with the held-out features, then compare results with the held-out labels to compute accuracy or error. section 4.5.3
- Keep evaluation examples uncontaminated: never place the query or gold answer being tested into the model's prompt history. section 6.4
Drug-price prediction
For 250,000 drug records, an 80/20 split assigns 200,000 rows to training and 50,000 to evaluation. The trained model predicts prices from unseen features, and each prediction is compared with its actual price; predicting 100 for an actual 110 represents a 10% error. section 4.5.2 section 4.5.3
Contamination-resistant SQL evaluation
A SQL generator receives few-shot examples from a separate frozen pool rather than the evaluation dataset. Retrieval filters by database and excludes the gold SQL under test, preventing inflated results when datasets overlap. section 6.4
Figure 4.5 The test/train split of the data.

Visualizes how rows become separate training and evaluation datasets after shuffling and proportion-based splitting. section 4.5.2








































