Fixed ratio

As explained in Machine Learning System Design

A split-update procedure that preserves the ratio of training data to validation data as new labeled data arrives.

When labeled images or text arrive irregularly and older examples remain relevant, split each newly labeled portion so the training-to-validation ratio stays unchanged. section 7.4

Use the split to keep evaluation meaningful as the dataset grows:

• Avoid placing new labels only in training, which can improve metrics simply because the model has more data. section 7.4
• Avoid placing new labels only in validation, which can leave the model without exposure to newly appearing patterns. section 7.4
• Split the added labels between both datasets using the existing proportion, then retrain and evaluate the model. section 7.4

Figure 7.7 Common options for updating train/validation sets. The light data blocks are used for training, while the dark ones are used for validation.

Compare fixed-ratio updating with fixed-shift and fixed-set alternatives through the train/validation block patterns. section 7.4

How Machine Learning System Design covers it

How deep it goes

Explains the split rationale and procedure, but does not detail sampling algorithms or ratio calculations.

Examples

Uses newly labeled images or text and contrasts adding labels only to training, only to validation, or to both.

What it assumes

Assumes familiarity with training and validation datasets, while explaining why newly labeled data should enter both.

What you can do after

Apply fixed-ratio updating when expanding labeled datasets without making evaluation depend on which split receives new examples.

How it connects

Connects dataset updating with training data, validation data, model retraining, and fixed-shift and fixed-set validation.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Related concepts

• evaluation datasetconfigures · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.