liveBook · Manning

Collection and purpose

As explained in AI Governance

Collection and purpose is a privacy-risk dimension concerning what data enters an AI system and whether its use remains limited to a stated purpose.

Collection and purpose asks what personal data enters an AI system and whether its use is lawful and expected. Unlike a single database, a GenAI system may combine fine-tuning data, retrieval content, prompts, and conversation logs. Separately cleaning each source may not prevent the same person from becoming identifiable across them. section 5.0 section 5.1

Treat data flows as an engineering responsibility:

  • Review what enters fine-tuning, retrieval-augmented generation, and conversation logging pipelines; organizations integrating models remain responsible for those flows. section 5.1
  • Anonymize or pseudonymize fine-tuning and retrieval data, set strict retention policies, explain data uses, offer opt-in or opt-out choices, and disable vendor prompt logging or training where available. section 5.1

Cross-source identification

Medical notes stripped of names may become identifying when a model reads them alongside chat logs, support tickets, and code containing the same person’s details. section 5.1

Table 5.1 Traditional Privacy vs GenAI Privacy
Privacy Topic
Traditional Application
GenAI Application
GenAI-specific Mitigations
Collection & Purpose
Only necessary fields are collected from the user or re-used from existing datasets. Lawful basis tied to collection context.
Training sets scraped from the web include people who never dealt with the vendor. A search engine listing can be removed on request; text absorbed into model weights cannot. Vendors may process user data by default for model improvement.
Input filters: anonymize or pseudonymize finetune/RAG data, apply strict retention policies, tell users how their data is used (e.g. to give AI-generated responses, train certain models) and offer opt-outs or opt-ins, opt-out of vendor logging/training of prompts and data.
Storage & Memorization
Records live in a database; deleting means removing a row.
Personal details can get “baked in” as patterns in model weights, embeddings, or vector databases.Models can spit out memorized names or documents; embeddings reveal hidden details
Encrypt and lock down embeddings/vector DBs, delete logs after a short period, test models for leakage
Output Integrity
Software gives fixed results you can trace to inputs. Data leakage typically results from misconfigurations or insider access. ML model misclassifications can be debugged.
Models can make up personal facts and false claims about people (defamation) non-deterministically (e.g. make guesses of user traits from context), can disclose sensitive data without being asked, LLM is a black box and debugging is difficult.
Output filters, isolate project/session, red-team for personal data leaks and hallucinations
User Rights & Governance
Users are offered tools to see, correct, or delete their data.
Hard to erase or fix info once it’s spread across billions of model parameters.
Reduce sensitive data in training (E.g. differential privacy/anonymization), clear opt-outs from model training, vendor contracts with delete options (retention, deletion upon ending the contract, delete if a user leaves the organization, etc.), model retraining, Machine unlearning (once it becomes more mature)

Compare conventional collection limits and lawful basis with GenAI collection risks and proposed mitigations. section 5.1

How AI Governance covers it

How deep it goes
Explains cross-source identification risks and lists practical controls, without detailing their implementation.
Examples
Uses combined medical notes, chat logs, support tickets, and code to illustrate cross-source identification.
What you can do after
Operate/configure data pipelines with anonymization, retention limits, user notices, consent choices, and vendor logging controls.
How it connects
Connects collection and purpose with storage and memorization, output integrity, and user rights and governance.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.