liveBook · Manning

Chapter 5: Privacy

What you will learn in chapter 5 of AI Governance

The chapter introduces how GenAI expands privacy risks, organizes them into four pillars, and emphasizes evidence of how personal data is handled.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The chapter introduces how GenAI expands privacy risks, organizes them into four pillars, and emphasizes evidence of how personal data is handled.

Key terms: personal data, privacy breach, prompts and logs, retrieval indexes, model or adapter parameters, stored outputs, vendor systems, backups, Collection and Purpose, Storage and Memorization

5.1 The Four Pillars of GenAI Privacy

This section explains why GenAI expands the privacy work required by familiar data-protection principles and organizes the resulting risks into four pillars.

Key terms: purpose, transparency, individual rights, model weights, embedding stores, retrieval pipelines, collection and purpose, anonymization, secondary use, valid legal basis

5.2 Collection and Purpose

This section defines collection broadly and shows why gathering more personal data than a GenAI task needs creates privacy risks that can be difficult to undo.

Key terms: collection, pretraining datasets, fine tuning corpora, retrieval indexes, user prompts, feedback signals, ClaimAssist, vehicle telematics data, data collection, training pipeline

5.2.1 No Valid Legal Basis

This subsection explains why using personal data for AI requires a valid, purpose-compatible legal basis and why legal permission alone may not preserve user trust.

Key terms: GDPR Article 6, lawful basis, consent, contractual necessity, legitimate interest, foundation models, web scrapes, public data, model training, data controllers

5.2.2 Overcollection

This section explains why GenAI projects must limit personal data collection, why removing identifiers may not make data or models anonymous, and how to reduce and test the remaining privacy risk.

Key terms: lawful basis, data minimization, over-collection, RAG, anonymization, re-identification, indirect identifiers, personal data, generalization, aggregation

5.2.3 Vendor’s Inappropriate Use of Organizational Data

This section explains the privacy risks of vendors’ secondary use of organizational GenAI data and how to limit, verify, and contain that exposure.

Key terms: secondary use, prompts and outputs, fine-tune, retrain, opt out, no training, retention limits, audit trails, data flow diagrams, logs

5.2.4 Adoption Model Differences

This section compares how SaaS use, API integration, and model hosting change an organization’s control over personal data and its privacy responsibilities.

Key terms: SaaS, PII redactor/anonymization, Data Loss Prevention, vendor-hosted RAG, secondary use, API, pre- and post-processing pipelines, minimization, central API gateway, local RAG

5.3 Storage and Memorization

This section uses a ClaimAssist leak to show that personal data may be learned by a model or persist across many parts of a GenAI system.

Key terms: RAG system, fine-tuning dataset, memorization, deletion request, retrieval documents, embeddings, prompt and output logs, model or adapter parameters

5.3.1 Memorization and Membership Inference

The section explains how generative models can expose training data or infer private facts about people, then presents baseline and advanced ways to reduce those risks.

Key terms: memorization, training corpus, training examples, membership inference, training set, attribute inference, anonymization, private attributes, output filters, canary phrases

5.3.2 Insufficient deletion of embeddings

This section explains why RAG embeddings can expose personal data and how to classify, protect, test, and reliably delete them.

Key terms: embeddings, vector store, RAG, reconstruction, attribute inference, personal data, GDPR, sensitive sources, provenance, retention periods

5.3.3 Insufficient Deletion of (Meta)data

This section explains why deleting personal data from a GenAI system requires finding copies and transformed forms beyond the primary application.

Key terms: erasure, data’s entire lifecycle, personal data, prompts and outputs, embeddings and indexes, memory features, fine-tuned model weights

5.3.4 Adoption Model Differences

This section compares where storage and memorization risks sit—and who can address them—across SaaS, API integrator, and model-hoster adoption models.

Key terms: SaaS, logs, RAG indexes, deletion cadence, API integrator, embeddings, time-to-live, automated purges, model hosters, model weights

5.4 Output Integrity

This section explains why confident but false GenAI outputs about real people are a privacy issue, especially when they affect significant decisions.

Key terms: output integrity, hallucinated, personal data, GDPR, significant decisions

5.4.1 Hallucinations and Defamations

This section explains how plausible AI-generated falsehoods about real people create privacy and reputational harms, and why the stakes rise when those falsehoods inform consequential decisions.

Key terms: right to rectification, fairness principle, special category data, offence data, Section 230, disclaimers, defamatory content, automated decision-making, Article 22, human intervention

5.4.2 Overreliance and Automated Decision Making

The section explains how trust in AI outputs can turn nominal human review into automated decision-making, and why consequential decisions need meaningful oversight, especially when agents can act autonomously.

Key terms: automation bias, overreliance, hallucinations, GDPR Article 22, meaningful human oversight, decision support, human in the loop, Consumer Financial Protection Bureau (CFPB), EEOC, legal accountability

5.4.3 Adoption Model Differences

This section compares output-integrity controls across SaaS, API-integrator, and model-hoster deployments and explains why human judgment remains necessary in all three.

Key terms: SaaS deployment, API integrator, model hoster, guardrail tools, human review, retrieval-augmented generation, document-level access control, retrieval-layer controls, output filters, vector search

5.5 User Rights & Governance

Elena’s access request shows how privacy rights become difficult to fulfill when personal data appears in embeddings and model weights, even though regulators continue to enforce those rights.

Key terms: Data Subject Access Request, Right of Access, Right of Transparency, Right of Erasure, RAG index, text embeddings, vector store, fine-tuning dataset, model weights, residual influence

5.5.1 Making models forget and correct

This section explains why trained models are hard to make forget or correct personal information and what organizations can do while reliable machine unlearning remains unavailable.

Key terms: machine unlearning, retraining, parameters, erasure, rectification, hallucinations, Everalbum, minimize personal data, deletion-aware pipelines, training examples

5.5.2 Data Subject Access Requests (DSAR)

This section explains why data subject access requests are difficult for generative AI and how organizations can respond using records they can retrieve while being candid about what they cannot extract from model weights.

Key terms: right of access, right to know, controllers, access requests, model weights, parameters, extraction, access rights gap, retrieval-augmented generation (RAG), embeddings

5.5.3 Transparency

This section explains why privacy transparency requires disclosing AI involvement and giving people understandable, traceable reasons for consequential AI-influenced outcomes.

Key terms: transparency, disclosure, informed consent, consequential decisions, explanations, adverse-action notices, traceability

5.5.4 Adoption Model Differences

This section compares how SaaS users, API integrators, and model hosters honor the same privacy rights through different technical and organizational controls.

Key terms: accountability, technical control, user rights, adoption model, SaaS, vendor due diligence, deletion, export, opt-out, transparency

5.6 Agentic AI and Privacy

This section explains how an agent’s autonomous actions can expand privacy risks beyond those created by a model’s responses.

Key terms: agentic AI, autonomous, goal-driven workflows, Tool Use, Persistent Memory, ClaimAssist 3.0, tool calls, purpose limitation, transparency, storage limitation

5.6.1 Purpose Limitation (and Data Minimization) Under Strain

This section explains how autonomous agents can expand the purposes and volume of personal-data processing, and how runtime controls can keep tool use within approved privacy boundaries.

Key terms: purpose limitation, mission creep, dynamic purpose creep, persistent memory, data exfiltration, MCP, dynamic tool discovery, rug pull, sub-processor, record of processing activities

5.6.2 Lawful Basis Challenges for Autonomous Processing

This section explains why autonomous agent processing needs a valid lawful basis for each use of personal data and how organizations can check that basis as agents act.

Key terms: Article 6 lawful basis, consent, specific, informed, legitimate interests, contractual necessity, Legitimate Interest Assessment (LIA), special category data, explicit consent, usage policies

5.6.3 Storage Limitation and Erasure Rights in an Agent Context

This section explains why agents’ persistent and distributed memory complicates storage limitation and erasure, and how to design retention, deletion, and visibility into agent data.

Key terms: storage limitation, right to erasure, persistent memory, distributed memory, purpose limitation, working memory, execution traces, longer term memory, fine tuning, machine unlearning

5.6.4 Governance and Operational Challenges in the Long Term

The section explains how to govern agents’ changing behavior and data flows through continuous compliance monitoring, human oversight, and documentation of permitted and actual tool use.

Key terms: non-deterministic behavior, continuous adaptation, data flows, AI observability, logs, metrics, traces, audit trail, alerts, human oversight

5.6.5 Summary of Privacy Risks and Mitigations in Agentic AI

This section summarizes why autonomous agents amplify privacy risks and why organizations must strengthen governance without relaxing GDPR obligations.

Key terms: agentic AI, GDPR, purpose limitation, lawful basis, transparency, data minimization, privacy by default, logging, oversight, dynamic risk assessments

5.7 Summary

This section summarizes where GenAI creates privacy risks and why deployment choices, verifiable evidence, and privacy-by-design controls matter.

Where this is used later

Concepts explained in this chapter

← Chapter 4: Securing GenAI Chapter 6: Ethical and Trustworthy GenAI →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.