liveBook · Manning

Memorization

As explained in LLM Customization and Fine-Tuning

A training outcome in which a model learns training examples without correspondingly improving its ability to generalize to new examples.

Memorization occurs when a model fits training examples without a comparable gain on new examples. A widening gap between falling training loss and nearly unchanged evaluation loss is a warning sign. section 2.8

A 40-example fine-tune

After 20 LoRA steps—about four passes over 40 IT support examples—training loss fell from 2.94 to 1.69, while evaluation loss moved from 1.98 to 1.87. That pattern is expected at this scale: the run demonstrates that the pipeline works, but not that the model learned behavior it can carry to new tickets. section 2.8

Memorization can also expose training data: personally identifiable information should be removed or anonymized unless its use is justified and legally permitted. section 5.11

Listing 2.5 SFTTrainer log over the 20 quickstart steps
  {'loss': 2.9427, 'grad_norm': 1.630, 'learning_rate': 0.000178, 'epoch': 1.0}
  {'loss': 1.9814, 'grad_norm': 0.453, 'learning_rate': 0.000122, 'epoch': 2.0}
  {'eval_loss': 1.9761, 'epoch': 2.0}
  {'loss': 1.7807, 'grad_norm': 0.527, 'learning_rate': 0.000067, 'epoch': 3.0}
  {'loss': 1.6865, 'grad_norm': 0.427, 'learning_rate': 0.000011, 'epoch': 4.0}
  {'eval_loss': 1.8741, 'epoch': 4.0}
  {'train_runtime': 78.45, 'train_samples_per_second': 2.04, 'train_loss': 2.0978, 'epoch': 4.0}

Compare the training and evaluation losses across the 20 steps. section 2.8

How LLM Customization and Fine-Tuning covers it

How deep it goes
Interprets a worked loss pattern, but defers evaluation decisions at production scale.
Examples
Uses a 40-example IT support fine-tune and a privacy warning about personally identifiable information.
What you can do after
Evaluate whether falling training loss is outpacing evaluation improvement, and recognize data leakage as a privacy concern.
How it connects
Connects memorization to fine-tuning, training loss, evaluation loss, and data privacy.

Also explained in

Compare how 6 books explain memorization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.