Personal data can persist after a prompt or dataset has been used. Unlike a conventional database record, it may be encoded in model weights, embeddings, or retrieval indexes, making deletion difficult to verify. section 5.1
Check the system’s full data path for:
- Training data that a model may reproduce verbatim. section 5.1
- Embedding databases and retrieval indexes that may retain or expose personal details. section 5.1
- Prompts, responses, and backups across vendor systems and organizational infrastructure. section 5.0
Memorized training data
Researchers prompted a model to repeat a word hundreds of times and extracted more than 10,000 verbatim text fragments, some containing names and contact details. section 5.1
Treat deletion as a verification problem, not just a record-removal task. Maintain auditable data pipelines and stress-test whether personal information persists in less visible layers. Controls and evidence should account for SaaS, API integrations, and self-hosted models. section 5.0 section 5.1
Table 5.1 Traditional Privacy vs GenAI Privacy
|
Privacy
Topic
|
Traditional
Application
|
GenAI
Application
|
GenAI-specific
Mitigations
|
|
Collection & Purpose
|
Only
necessary fields are collected from the user or re-used from existing
datasets. Lawful basis tied to collection context.
|
Training
sets scraped from the web include people who never dealt with the vendor. A
search engine listing can be removed on request; text absorbed into model
weights cannot. Vendors may process user data by default for model
improvement.
|
Input
filters: anonymize or pseudonymize finetune/RAG data, apply strict retention
policies, tell users how their data is used (e.g. to give AI-generated
responses, train certain models) and offer opt-outs or opt-ins, opt-out of
vendor logging/training of prompts and data.
|
|
Storage & Memorization
|
Records live
in a database; deleting means removing a row.
|
Personal
details can get “baked in” as patterns in model weights, embeddings, or
vector databases.Models can spit out memorized names or documents; embeddings
reveal hidden details
|
Encrypt and
lock down embeddings/vector DBs, delete logs after a short period, test
models for leakage
|
|
Output Integrity
|
Software
gives fixed results you can trace to inputs. Data leakage typically results
from misconfigurations or insider access. ML model misclassifications can be
debugged.
|
Models can
make up personal facts and false claims about people (defamation)
non-deterministically (e.g. make guesses of user traits from context),
can disclose sensitive data without being asked, LLM is a black box and
debugging is difficult.
|
Output
filters, isolate project/session, red-team for personal data leaks and
hallucinations
|
|
User Rights & Governance
|
Users are
offered tools to see, correct, or delete their data.
|
Hard to
erase or fix info once it’s spread across billions of model parameters.
|
Reduce
sensitive data in training (E.g. differential privacy/anonymization), clear
opt-outs from model training, vendor contracts with delete options
(retention, deletion upon ending the contract, delete if a user leaves the
organization, etc.), model retraining, Machine unlearning (once it becomes
more mature)
|
Compare conventional record deletion with GenAI persistence in model weights and other data stores. section 5.1
