A poisoning attack corrupts a model before deployment by inserting malicious or misleading information into training or post-training data. The model may behave normally in most situations while producing harmful outputs in targeted contexts. section 5.3 section 10.10
The attack exploits the gap between collecting data and training on it. An adversary can alter frequently crawled websites shortly before a dataset refresh or modify periodically captured sources before a known snapshot date. section 5.3
Poisoning creates several security concerns:
- Small amounts of contaminated data may introduce subtle vulnerabilities or cause failures that appear only in particular situations. section 5.3
- Attackers have used poisoned datasets to bypass phishing filters or insert backdoors into code-generation models. section 10.10
- Tool-augmented systems raise the stakes because compromised models may influence code execution, web browsing, API calls, or other real-world actions. section 10.10
Spam-filter misclassification
A spam filter trained on clean labeled examples learns to distinguish spam from legitimate email. Poisoned training examples can distort that boundary, causing malicious messages to be classified as legitimate. section 5.3
Figure 5.4 Training a spam filter on clean data: the model learns to classify emails as spam or ham based on labeled examples.
Use the clean-data diagram to compare ordinary classification training with the later poisoned-data outcome. section 5.3
Figure 5.5 Training on poisoned data causes spam misclassification: malicious inputs distort the model’s behavior, leading it to incorrectly label spam as legitimate.
Use the poisoned-data diagram to see how malicious inputs can make a spam filter label spam as legitimate. section 5.3
