Training data extraction is an attack that repeatedly queries a trained language model to recover content memorized from its training data. The recovered material may include personally identifiable information (PII), contact details, medical information, conversations, code, or identifiers—even when the information appeared rarely or only once. section 3.4 section 10.9
The risk is greatest when models trained on proprietary or sensitive data are publicly deployed. Extraction raises ethical and legal concerns because ordinary-looking prompts can expose information that people did not expect a model to reproduce. section 3.4 section 10.9
Important findings include:
- Larger models showed greater susceptibility to memorizing and revealing rare training sequences than smaller models. section 3.4
- Safety and alignment techniques reduce neither the possibility nor the scale of leakage completely; a divergence attack extracted data more easily than expected. section 3.4
- Privacy exposure can occur both through model memories and through user inputs or chat histories surfaced to others. section 10.9
GPT-2 extraction demonstration
A 2021 research collaboration extracted hundreds of memorized GPT-2 sequences, including names, phone numbers, email addresses, instant messages, code, and universally unique identifiers. A 2023 study reported that gigabytes could be extracted from open- and closed-source language models. section 3.4
