Memorization

Q: How deeply does Evolutionary Deep Learning: Genetic algorithms and neural networks cover memorization?

Explains causes, training signals, and interventions through a CNN experiment, but stops short of formal theory.

As explained in Evolutionary Deep Learning: Genetic algorithms and neural networks

Model memorization is the phenomenon in which a model fits particular training examples without learning patterns that transfer to new data.

Model memorization occurs when a network fits particular training examples instead of learning patterns that transfer to new data. It becomes more likely when a large model encounters a small or reduced training set. section 1.3.1 section 7.1.1

A memorizing model can approach perfect training accuracy while validation or test performance declines. This reveals specialization rather than generalization, although specialization can be useful when identifying a specific dataset is the goal. section 7.1.1

Inspect memorization through these practical signals and interventions:

Compare training and validation accuracy and loss rather than relying on training accuracy alone. section 7.1.1
Use dropout to deactivate a percentage of nodes during each training pass, encouraging a more general network at added computational and size cost. section 1.3.1
Use pooling and batch normalization to reduce model size and improve generalization, while checking whether memorization remains. section 7.1.1

Single-layer CNN

A CNN with 64 filters, a 3×3 kernel, flattening, and dense layers contains over six million parameters. On a reduced dataset, it reaches almost 100% training accuracy while validation or test accuracy falls. Adding pooling and batch normalization reduces the model to about 1.6 million parameters and produces improving validation accuracy with decreasing loss, though some memorization remains. section 7.1.1

How Evolutionary Deep Learning: Genetic algorithms and neural networks covers it

Cover of Evolutionary Deep Learning: Genetic algorithms and neural networks

liveBook manning.com

Read Evolutionary Deep Learning: Genetic algorithms and neural networks on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

The AI Pocket BookA behavior where a model stores frequent patterns or solutions from training data rather than learning a general rule.
liveBook manning.com

Introduction to Generative AI, Second EditionModel memorization is the retention of specific training examples or sensitive information in a trained model's parameters or behavior.
liveBook manning.com

Sutskever's ListMemorization is a model's retention of specific training examples or their details rather than learning patterns that generalize.

Compare how 4 books explain memorization →

Related concepts

generalizationcontrasts with · Generalization is the extension of a pattern, model, solution, or conclusion beyond the specific examples or cases from which it was derived.
training datasetproduced by · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.
dropoutaddressed by · Dropout is a neural-network regularization technique that randomly suppresses units, connections, or activations during training to reduce overfitting.
fine-tuningproduced by · Fine-tuning is adapting a pretrained machine-learning model by training it further on targeted data or objectives for a task, domain, or behavior.
trainingcontrasts with · Training is a process of developing or improving a person’s or model’s capabilities through instruction, experience, or adjustment using data.

This explainer was generated by Manning's knowledge graph from the text of Evolutionary Deep Learning: Genetic algorithms and neural networks. The section links point to the parts of the book it draws on.