Fixed set

As explained in Machine Learning System Design

A split-update procedure that evaluates models on an unchanged reference dataset across modeling iterations.

A fixed set is an unchanged “golden set” used to evaluate successive models. Keeping it constant preserves metric comparability across a long modeling period, even as the available training data grows. section 7.4

Use a fixed set when consistent comparison matters:

• Sample it before modeling, or manually select diverse hard cases and reference responses. section 7.4
• Do not update it by design; if it is extended later, treat the extension as a completely new benchmark. section 7.4
• Validate the whole pipeline, including the dataset, and run test-set inference the same way as production inference. section 7.4
• Save previous dataset and model versions when accurate side-by-side comparisons are required; data and model version-control tools can help. section 7.4

Figure 7.7 Common options for updating train/validation sets. The light data blocks are used for training, while the dark ones are used for validation.

Compare the fixed-set option with alternative ways to update training and validation data. section 7.4

How Machine Learning System Design covers it

How deep it goes

Explains construction, immutability, benchmark replacement, and pipeline-validation implications without detailing statistical theory.

Examples

Uses sampled or manually cherry-picked golden sets containing diverse hard cases and reference responses.

What it assumes

Assumes a modeling workflow with training, validation, test data, and model-version comparison.

What you can do after

Evaluate model versions consistently using an unchanged benchmark and production-matched inference pipeline.

How it connects

Connects validation datasets with training data, test-set inference, production inference, and data and model version control.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

• evaluation datasetuses · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.