liveBook · Manning

Membership inference attack

As explained in AI Governance

An attack that attempts to determine whether particular data was included in a model’s training data.

A membership inference attack probes a model to determine whether a particular person’s data was used in training. Membership can itself be sensitive: inclusion in a cancer registry or bankruptcy dataset may reveal information about the person, even if the model does not reproduce a record verbatim. section 5.3.1 section 5.2.2

An attacker asks indirect questions and looks for differences between the model’s responses to real facts and made-up ones. Those differences can provide clues about training-set membership. This differs from memorization, where training data reappears in an output, and from attribute inference, where a model predicts hidden traits from text. section 5.3.1

Sensitive registry membership

If a model was trained on a cancer registry, an attacker who infers that a person’s data was included may learn a sensitive fact about that person. The privacy failure does not require the model to disclose the person’s full record. section 5.3.1 section 5.2.2

Reduce exposure across the training and deployment lifecycle:

  • Filter sensitive material, remove duplicates, and limit access before using data for training. section 5.3.1
  • Where personal data is used for fine-tuning, account for membership inference in the model and actively test for memorization; prevent personal data from entering training when possible. section 5.3.4
  • Assess anonymity using measures such as filtering or pseudonymization, privacy-enhancing techniques during training, and testing for known attacks. Removing direct identifiers alone is not foolproof. section 5.2.2

How AI Governance covers it

How deep it goes
Explains indirect probing through differences in responses to real and made-up facts, without a detailed attack procedure.
Examples
Uses cancer registries, political dissident datasets, and bankruptcy records to illustrate sensitive membership.
What you can do after
Evaluate privacy exposure from training-set membership and identify relevant training-data and model-hosting safeguards.
How it connects
Distinguishes membership inference from memorization and attribute inference, and links it to training-data governance.

Also explained in

Compare how 2 books explain membership inference attack →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.