Hold-out validation

As explained in Deep Learning with R, Third Edition

Hold-out validation is an evaluation procedure that reserves a subset of available data for assessment without using it to fit the model.

Hold-out validation evaluates a model by reserving part of the available data, training on the remainder, and measuring performance on the reserved validation set. Keep a separate, untouched test set for final evaluation. section 5.2.1 section 5.5

A practical hold-out workflow is:

• Reserve a representative validation fraction and avoid redundant samples between training and validation data. section 6.2.2
• Train the model only on the training partition, then use validation metrics to guide modeling decisions and hyperparameter tuning. section 6.2.2
• After decisions are complete, retrain on the available non-test data and evaluate once on the never-seen test set. section 5.2.1

A simple split

num_validation_samples <- 10000 val_indices <- sample.int(nrow(data), num_validation_samples) validation_data <- data[val_indices, ] training_data <- data[-val_indices, ] model <- get_model() fit(model, training_data, ...) validation_score <- evaluate(model, validation_data, ...) # After tuning model <- get_model() fit(model, data, ...) test_score <- evaluate(model, test_data, ...) section 5.2.1

Figure 5.12 Simple hold-out validation split

A split diagram makes the separation between training, validation, and test data immediately visible. section 5.2.1

Hold-out validation is the simplest evaluation protocol and usually works when plenty of data is available. With little data, random splits may produce unstable or unrepresentative scores; use K-fold or iterated K-fold validation instead. section 5.2.1 section 6.2.2

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains the split, training loop, test separation, leakage risk, and small-data limitation, then points to K-fold alternatives.

Examples

Uses indexed data partitioning, model fitting, validation scoring, final retraining, and test evaluation.

What it assumes

Assumes an annotated dataset and chosen success metric while establishing training, validation, and test-set roles.

What you can do after

Apply it by splitting data, tuning against validation metrics, retraining after decisions, and reserving the test set for final evaluation.

How it connects

Connects training sets, validation sets, test sets, generalization, information leakage, hyperparameter tuning, and K-fold cross-validation.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionAn evaluation protocol that splits data once into separate subsets for training, validation, and/or testing.

Compare how 2 books explain hold-out validation →

Related concepts

• cross-validationalternative to · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
• validationkind of · The process of checking data, behavior, systems, models, or results against defined criteria to assess their correctness, suitability, or acceptability.
• evaluation datasetuses · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
• training datasetimplements · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.