How deeply does LLMs in Production cover validation loss?

Defines the metric and implements ten-batch averaging, but stops short of treating it as a representative performance measure.

Validation loss

As explained in LLMs in Production

Validation loss is the loss computed on held-out validation examples to assess a model during development.

Validation loss averages loss across validation batches to provide a practical development signal. It is useful for monitoring training, but it is not a complete representation of model performance. section 9.1.2

Estimating validation loss

A loss-evaluation function switches the model to evaluation mode, samples ten batches from the validation split, averages their losses, and then restores training mode.

Use validation loss with deliberate interpretation:

Treat the average as a convenient marker, not as a full description of how the model performs. section 9.1.2

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Deep Learning with R, Third EditionValidation loss is the loss value computed from a model's predictions on validation data.
liveBook manning.com

Deep Learning with PyTorch, Second EditionLoss computed on held-out validation data.
liveBook manning.com

Deep Learning with Python, Third EditionThe loss value computed on validation data to estimate how well a model generalizes beyond the training set.
liveBook manning.com

Generative AI in ActionValidation loss is a metric measuring a model’s prediction error on examples held out for validation.

Compare how 5 books explain validation loss →

Related concepts

objective functioncontrasts with · An objective function assigns values to candidate solutions, predictions, or parameter settings according to a criterion to be minimized or maximized.
generalizationmeasures · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
overfittingmeasures · Overfitting is a modeling condition in which a model fits patterns or idiosyncrasies in its training data too closely and consequently generalizes poorly to new data.
evaluation datasetmeasures · An evaluation dataset is a collection of examples or test cases, often with expected or judged outcomes, designated to assess a model or system against defined criteria.
scaling lawmeasured by · A scaling law is an empirical relationship describing how a system’s properties change as its scale or resource levels vary.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.