A model can reveal private traits even when it does not repeat personal data verbatim. Attribute inference predicts an unknown or sensitive attribute from available data or model outputs. It differs from membership inference, which asks whether a person’s data appeared in a training set. section 5.3.1
Profiling from anonymous posts
In a 2024 study of Reddit users, GPT-4 predicted demographics including age, location, sex, and income correctly on its first attempt 85% of the time; the correct answer appeared in its top three guesses 95% of the time. Removing obvious identifiers reduced location accuracy from 86% to 55%, but did not eliminate the inference risk. Writing cues such as slang, punctuation, and references could still support predictions. section 5.3.1
The attack also applies to embeddings. In a retrieval-augmented generation pipeline, an attacker may predict a private fact, such as a patient diagnosis, from an embedding without seeing the original text. This is distinct from reconstruction, which attempts to recover text from the vector. section 5.3.2
Treat inference as an operational privacy risk:
- Test model outputs for known attacks, including attribute inference and reconstruction, when assessing anonymity. section 5.2.2
- Classify embeddings by source and linkability. Treat embeddings from sensitive sources, such as patient records or HR files, as personal data. section 5.3.2
- Do not rely on removing obvious identifiers alone. Consider privacy-enhancing techniques during training and test whether individuals can still be identified indirectly. section 5.3.1 section 5.2.2
