liveBook · Manning

Evaluation dataset

As explained in Vibe Engineering

A held-out portion of data used to evaluate a trained machine-learning model on previously unseen inputs and targets.

An evaluation dataset is held-out data reserved for checking a trained model on inputs and targets it did not use during training. Comparing predictions with known labels reveals accuracy and prediction error. section 4.5.2 section 4.5.3

Use a disciplined split rather than treating every available row as training data:

  • Shuffle first when rows contain implicit ordering such as dates, categories, or alphabetical ranges. section 4.5.2
  • Allocate separate training and evaluation portions, such as 80/20, and retain features and labels for both portions. section 4.5.2
  • Run prediction with the held-out features, then compare results with the held-out labels to compute accuracy or error. section 4.5.3
  • Keep evaluation examples uncontaminated: never place the query or gold answer being tested into the model's prompt history. section 6.4

Drug-price prediction

For 250,000 drug records, an 80/20 split assigns 200,000 rows to training and 50,000 to evaluation. The trained model predicts prices from unseen features, and each prediction is compared with its actual price; predicting 100 for an actual 110 represents a 10% error. section 4.5.2 section 4.5.3

Contamination-resistant SQL evaluation

A SQL generator receives few-shot examples from a separate frozen pool rather than the evaluation dataset. Retrieval filters by database and excludes the gold SQL under test, preventing inflated results when datasets overlap. section 6.4

Figure 4.5 The test/train split of the data.

Visualizes how rows become separate training and evaluation datasets after shuffling and proportion-based splitting. section 4.5.2

How Vibe Engineering covers it

How deep it goes
Explains shuffling, split proportions, feature-label separation, prediction, accuracy comparison, and prompt contamination safeguards.
Examples
Uses drug-price prediction, medicine-risk classification, and SQL generation with frozen example queries.
What it assumes
Assumes features, labels, and model training; teaches the train/evaluation split and validation workflow explicitly.
What you can do after
Evaluate trained models, calculate prediction error, and detect contamination that would make test results artificially optimistic.
How it connects
Connects model training, supervised learning, model validation, data contamination, and few-shot prompting.

Also explained in

Compare how 41 books explain evaluation dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Vibe Engineering. The section links point to the parts of the book it draws on.