How deeply does Sutskever's List cover memorization?

Explains factors shaping retention and gives random-label results, without detailing a mechanism for how examples are stored.

Memorization

As explained in Sutskever's List

Memorization is a model's retention of specific training examples or their details rather than learning patterns that generalize.

Memorization is retaining specific training examples or their details instead of learning patterns that generalize. In language models, greater scale can increase the capacity to store and reproduce training text, while repetition, redundancy, and training duration also shape what is retained. section 6.2.2

Fitting random labels

In experiments reported in 2016, AlexNet achieved 99.82% training accuracy on CIFAR-10 with randomized labels, and 95.20% top-1 training accuracy on ImageNet with more than one million random labels. High training accuracy therefore need not mean a network learned meaningful structure. section 8.5.2

Memorization complicates evaluation: language-model training data may overlap with public benchmarks such as SQuAD and GLUE, making it hard to tell whether a benchmark was encountered during training. Such contamination can inflate scores, though it does not account for all gains. section 6.2.2

Table 8.1 Training and test accuracy of a smaller version of AlexNet on the CIFAR-10 dataset. The original AlexNet had 60 million parameters.

Model	Number of parameters	Labels	Random crop	Weight decay	Train accuracy	Test accuracy
AlexNet	1,387,786	true	yes	yes	99.90	81.22
		true	yes	no	99.82	79.66
		true	no	yes	100.0	77.36
		true	no	no	100.0	76.07
		random	no	no	99.82	9.86

Compare AlexNet’s training and test accuracy when trained on true versus randomized labels. section 8.5.2

How Sutskever's List covers it

liveBook manning.com

Read Sutskever's List on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

The AI Pocket BookA behavior where a model stores frequent patterns or solutions from training data rather than learning a general rule.
liveBook manning.com

Introduction to Generative AI, Second EditionModel memorization is the retention of specific training examples or sensitive information in a trained model's parameters or behavior.
liveBook manning.com

Evolutionary Deep Learning: Genetic algorithms and neural networksModel memorization is the phenomenon in which a model fits particular training examples without learning patterns that transfer to new data.

Compare how 4 books explain memorization →

Related concepts

generalizationcontrasts with · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
training datasetproduced by · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.
dropoutaddressed by · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
fine-tuningproduced by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
trainingcontrasts with · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of Sutskever's List. The section links point to the parts of the book it draws on.