Collection and purpose asks what personal data enters an AI system and whether its use is lawful and expected. Unlike a single database, a GenAI system may combine fine-tuning data, retrieval content, prompts, and conversation logs. Separately cleaning each source may not prevent the same person from becoming identifiable across them. section 5.0 section 5.1
Treat data flows as an engineering responsibility:
- Review what enters fine-tuning, retrieval-augmented generation, and conversation logging pipelines; organizations integrating models remain responsible for those flows. section 5.1
- Anonymize or pseudonymize fine-tuning and retrieval data, set strict retention policies, explain data uses, offer opt-in or opt-out choices, and disable vendor prompt logging or training where available. section 5.1
Cross-source identification
Medical notes stripped of names may become identifying when a model reads them alongside chat logs, support tickets, and code containing the same person’s details. section 5.1
Table 5.1 Traditional Privacy vs GenAI Privacy
|
Privacy
Topic
|
Traditional
Application
|
GenAI
Application
|
GenAI-specific
Mitigations
|
|
Collection & Purpose
|
Only
necessary fields are collected from the user or re-used from existing
datasets. Lawful basis tied to collection context.
|
Training
sets scraped from the web include people who never dealt with the vendor. A
search engine listing can be removed on request; text absorbed into model
weights cannot. Vendors may process user data by default for model
improvement.
|
Input
filters: anonymize or pseudonymize finetune/RAG data, apply strict retention
policies, tell users how their data is used (e.g. to give AI-generated
responses, train certain models) and offer opt-outs or opt-ins, opt-out of
vendor logging/training of prompts and data.
|
|
Storage & Memorization
|
Records live
in a database; deleting means removing a row.
|
Personal
details can get “baked in” as patterns in model weights, embeddings, or
vector databases.Models can spit out memorized names or documents; embeddings
reveal hidden details
|
Encrypt and
lock down embeddings/vector DBs, delete logs after a short period, test
models for leakage
|
|
Output Integrity
|
Software
gives fixed results you can trace to inputs. Data leakage typically results
from misconfigurations or insider access. ML model misclassifications can be
debugged.
|
Models can
make up personal facts and false claims about people (defamation)
non-deterministically (e.g. make guesses of user traits from context),
can disclose sensitive data without being asked, LLM is a black box and
debugging is difficult.
|
Output
filters, isolate project/session, red-team for personal data leaks and
hallucinations
|
|
User Rights & Governance
|
Users are
offered tools to see, correct, or delete their data.
|
Hard to
erase or fix info once it’s spread across billions of model parameters.
|
Reduce
sensitive data in training (E.g. differential privacy/anonymization), clear
opt-outs from model training, vendor contracts with delete options
(retention, deletion upon ending the contract, delete if a user leaves the
organization, etc.), model retraining, Machine unlearning (once it becomes
more mature)
|
Compare conventional collection limits and lawful basis with GenAI collection risks and proposed mitigations. section 5.1
