Sampling gap

As explained in Sutskever's List

The mismatch between the limited inputs represented in an evaluation or dataset and the much larger distribution of possible inputs.

A sampling gap arises because benchmarks cover only a thin slice of the inputs a modern model may encounter. New capabilities and failures can surface after deployment when real inputs fall outside that slice. Because language models learn from text shaped by what people write and repeat, rare or underdescribed patterns remain weaker. section 6.2.1

Uneven performance

A model may handle multidigit arithmetic up to n digits but fail at n + 1, or retrieve information up to L tokens of context but slip at L + 1. These boundaries show why broad scores can conceal uneven capabilities. section 6.2.1

Figure 6.4 A model will learn some tasks more easily than others. Easy tasks are frequent with dense data, while hard tasks are infrequent with sparse data.

Shows how frequent tasks with dense data can be easier to learn than infrequent tasks with sparse data. section 6.2.1

How Sutskever's List covers it

How deep it goes

Defines the gap and traces its effects through training-data coverage and uneven performance, without quantifying it.

Examples

Contrasts arithmetic that fails at n + 1 digits with retrieval that slips beyond L tokens of context.

What you can do after

Understand why benchmark scores may not predict performance on inputs encountered after deployment.

How it connects

Connects benchmark evaluation to training distributions and next-token prediction.

liveBookmanning.com

Read Sutskever's List on liveBook Get the book at manning.com

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.