liveBook · Manning

Attribute inference

As explained in AI Governance

Attribute inference is an attack that infers sensitive or otherwise unknown attributes about an individual from available data or model outputs.

A model can reveal private traits even when it does not repeat personal data verbatim. Attribute inference predicts an unknown or sensitive attribute from available data or model outputs. It differs from membership inference, which asks whether a person’s data appeared in a training set. section 5.3.1

Profiling from anonymous posts

In a 2024 study of Reddit users, GPT-4 predicted demographics including age, location, sex, and income correctly on its first attempt 85% of the time; the correct answer appeared in its top three guesses 95% of the time. Removing obvious identifiers reduced location accuracy from 86% to 55%, but did not eliminate the inference risk. Writing cues such as slang, punctuation, and references could still support predictions. section 5.3.1

The attack also applies to embeddings. In a retrieval-augmented generation pipeline, an attacker may predict a private fact, such as a patient diagnosis, from an embedding without seeing the original text. This is distinct from reconstruction, which attempts to recover text from the vector. section 5.3.2

Treat inference as an operational privacy risk:

  • Test model outputs for known attacks, including attribute inference and reconstruction, when assessing anonymity. section 5.2.2
  • Classify embeddings by source and linkability. Treat embeddings from sensitive sources, such as patient records or HR files, as personal data. section 5.3.2
  • Do not rely on removing obvious identifiers alone. Consider privacy-enhancing techniques during training and test whether individuals can still be identified indirectly. section 5.3.1 section 5.2.2

How AI Governance covers it

How deep it goes
Defines inference from text and embeddings and reports evaluation results, without detailing an attack implementation.
Examples
Uses Reddit demographic predictions and a hypothetical patient diagnosis inferred from embeddings.
What you can do after
Evaluate privacy exposure by testing models for inference attacks and classifying embeddings by sensitivity and linkability.
How it connects
Distinguishes membership inference and reconstruction from attribute inference, and links the attack to embeddings and retrieval-augmented generation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.