liveBook · Manning

Memorization

As explained in AI Governance

The retention of specific training-data details by a model, potentially allowing it to reproduce them.

A model can retain specific details from its training data and reproduce them in response to a prompt. This memorization creates privacy and intellectual-property risks when outputs reveal personal records, confidential material, or copyrighted content. section 5.3.1 section 6.6.3

Unlike a database record, a learned detail may be embedded in model weights rather than stored in an easily deleted location. Repeated training images are particularly likely to be memorized; carefully crafted prompts can sometimes extract or closely reconstruct them. Recognizable resemblance alone, however, is not proof of an exact copy. section 5.1 section 5.3.1

A claim that resurfaced

ClaimAssist included a claimant’s name in a routine response. Investigation found her claim both in a retrieval index without a retention limit and in a fine-tuning dataset. A red-team engineer later extracted her phone number by prompting with accident details. The case required investigating both stored retrieval data and learned model behavior. section 5.3

Memorization is distinct from other ways data persists: logs, backups, embeddings, retrieval indexes, and vendor systems may hold copies even when the model has not learned them. An output that reproduces retrieved material can also raise licensing concerns, so investigation should trace both training and retrieval paths. section 5.3 section 6.6.3

Figure 5.1 Where personal data can live in a GenAI system. The exact boxes depend on the adoption model, but access, correction, deletion, and retention controls rarely stop at the main application database.

Use the data-location map to scope access, retention, and deletion checks beyond the application database. section 5.3

Reduce and test the exposure across the data lifecycle:

  • Collect only data needed for the purpose, and prevent personal data from entering training when possible. section 5.2 section 5.3.4
  • Test models for memorization, including adversarial attempts to elicit sensitive training details. section 5.3.4 section 5.3
  • Trace copies through fine-tuning datasets, model artifacts, retrieval stores, logs, backups, and vendors; maintain auditable pipelines and stress-test deletion. section 5.3 section 5.1
  • Review licensing for material supplied through retrieval or fine-tuning; provider safeguards for base-model data may not cover it. section 6.6.3

How AI Governance covers it

How deep it goes
Explains extraction risks and lifecycle controls, but stops short of a model-internals mechanism or a deletion procedure.
Examples
Uses extracted text fragments, reconstructed training images, and an insurance claim exposed through retrieval and fine-tuning.
What you can do after
Evaluate memorization exposure by mapping data locations, testing for leakage, and considering deletion and licensing constraints.
How it connects
Connects memorization to membership inference, retrieval-augmented generation, fine-tuning, data minimization, and deletion.

Also explained in

Compare how 6 books explain memorization →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.