Validation loss

As explained in Generative AI in Action

Validation loss is a metric measuring a model’s prediction error on examples held out for validation.

Validation loss measures a fine-tuned model’s prediction error on a validation set: data held aside to evaluate performance on new or unseen examples. section 9.3.4

Use validation loss to interpret training progress:

• Calculate it with the same cost function as training loss by summing errors across validation examples. section 9.3.4
• Measure it usually after each epoch, alongside training loss and token-accuracy metrics. section 9.3.4
• Prefer decreasing training and validation loss; persistent training improvement without comparable validation improvement can indicate overfitting and poor generalization. section 9.3.4

Figure 9.3 Training loss when fine-tuning GPT-3.5 Turbo

Inspect the fine-tuning loss graph to compare loss values across training steps and observe whether performance improves. section 9.3.4

How Generative AI in Action covers it

How deep it goes

Defines the metric, its cost-function calculation, epoch timing, and relationship to overfitting without mathematical derivation.

Examples

Uses a GPT-3.5 Turbo fine-tuning loss graph showing loss values across training steps.

What it assumes

Assumes a validation dataset and introduces validation loss alongside training loss and token-accuracy metrics.

What you can do after

Evaluate fine-tuning progress and identify potential overfitting by comparing training and validation loss.

How it connects

Connects validation loss with training loss, validation datasets, cost functions, epochs, token accuracy, overfitting, and generalization.

liveBookmanning.com

Read Generative AI in Action on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionValidation loss is the loss value computed from a model's predictions on validation data.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionLoss computed on held-out validation data.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe loss value computed on validation data to estimate how well a model generalizes beyond the training set.
• 

  liveBookmanning.com
  LLMs in ProductionValidation loss is the loss computed on held-out validation examples to assess a model during development.

Compare how 5 books explain validation loss →

Related concepts

• objective functioncontrasts with · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
• generalizationmeasures · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
• overfittingmeasures · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
• evaluation datasetmeasures · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
• scaling lawmeasured by · A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.

This explainer was generated by Manning's knowledge graph from the text of Generative AI in Action. The section links point to the parts of the book it draws on.