liveBook · Manning

Validation loss

As explained in Timeless Algorithms, The Foundational Ideas

Held-out loss is the value of a loss function evaluated on observations excluded from model training.

Validation loss, or held-out loss, is the value of a loss function evaluated on observations excluded from training. section 13.2.1

For neural language models, held-out loss measures predictive performance. It declined as parameter counts, training-data volume, and compute increased, following power laws within the range studied. section 13.2.1 section 13.2.2 section 13.2.3

Lower loss indicates reduced predictive uncertainty, but does not guarantee reasoning, factual accuracy, or reliability; continued reductions also require progressively greater resources. section 13.6

How Timeless Algorithms, The Foundational Ideas covers it

How deep it goes
Defines held-out loss and relates its decline to parameters, training data, and compute through power laws, without guaranteeing broader reliability.
Examples
Tracks next-token prediction loss as parameter counts, training-token volumes, or compute increase.
What you can do after
Evaluate scaling behavior by comparing held-out loss as model parameters, training data, or compute increase within the studied range.
How it connects
Connects to model capacity, training data, compute, overfitting, generalization, and next-token prediction.

Also explained in

Compare how 7 books explain validation loss →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Timeless Algorithms, The Foundational Ideas. The section links point to the parts of the book it draws on.