liveBook · Manning

Privacy pillar

As explained in AI Governance

A privacy pillar is one of the organizing dimensions used to describe where privacy risks emerge across the generative AI lifecycle.

A privacy pillar is a dimension for organizing where privacy risks emerge across the generative AI lifecycle. Database and log reviews alone may miss risks in model weights, embedding stores, retrieval pipelines, and generated outputs. section 5.1

Collection and purpose highlights risks in data entering or being reused by AI systems:

  • Combining datasets can make people identifiable even when individual sources have been cleaned. section 5.1
  • Prompts and responses may be processed for model improvement, creating secondary-use concerns. section 5.1

Training-data extraction

Researchers prompted ChatGPT (gpt-3.5-turbo) to repeat a word until it began outputting training data, including personal details. section 5.1

Table 5.1 Traditional Privacy vs GenAI Privacy
Privacy Topic
Traditional Application
GenAI Application
GenAI-specific Mitigations
Collection & Purpose
Only necessary fields are collected from the user or re-used from existing datasets. Lawful basis tied to collection context.
Training sets scraped from the web include people who never dealt with the vendor. A search engine listing can be removed on request; text absorbed into model weights cannot. Vendors may process user data by default for model improvement.
Input filters: anonymize or pseudonymize finetune/RAG data, apply strict retention policies, tell users how their data is used (e.g. to give AI-generated responses, train certain models) and offer opt-outs or opt-ins, opt-out of vendor logging/training of prompts and data.
Storage & Memorization
Records live in a database; deleting means removing a row.
Personal details can get “baked in” as patterns in model weights, embeddings, or vector databases.Models can spit out memorized names or documents; embeddings reveal hidden details
Encrypt and lock down embeddings/vector DBs, delete logs after a short period, test models for leakage
Output Integrity
Software gives fixed results you can trace to inputs. Data leakage typically results from misconfigurations or insider access. ML model misclassifications can be debugged.
Models can make up personal facts and false claims about people (defamation) non-deterministically (e.g. make guesses of user traits from context), can disclose sensitive data without being asked, LLM is a black box and debugging is difficult.
Output filters, isolate project/session, red-team for personal data leaks and hallucinations
User Rights & Governance
Users are offered tools to see, correct, or delete their data.
Hard to erase or fix info once it’s spread across billions of model parameters.
Reduce sensitive data in training (E.g. differential privacy/anonymization), clear opt-outs from model training, vendor contracts with delete options (retention, deletion upon ending the contract, delete if a user leaves the organization, etc.), model retraining, Machine unlearning (once it becomes more mature)

Compare traditional privacy practices with generative AI risks and collection-focused mitigations. section 5.1

How AI Governance covers it

How deep it goes
Defines four organizing dimensions but details collection and purpose through data aggregation, training-data leakage, and secondary use.
Examples
Uses combined datasets, vendor prompt processing, and a ChatGPT training-data extraction attack as examples.
What you can do after
Apply input filtering, data anonymization or pseudonymization, retention limits, user notices, and vendor logging or training opt-outs.
How it connects
Connects collection and purpose to training data, retrieval-augmented generation, data lineage, and model improvement.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.