Poisoning attack
A poisoning attack deliberately corrupts data, a learning process, or model artifacts to compromise a machine-learning system’s behavior or outputs.
Introduction to Generative AI, Second Edition
Data poisoning is an attack that inserts malicious or misleading examples into training data so a model learns harmful or exploitable behavior.
- How it is used
- Explains how attackers influence generative AI behavior by corrupting training or post-training data.
- How deep it goes
- Describes attack timing, dataset exposure, targeted failures, and broad consequences without detailing a complete defense procedure.
- Examples
- Uses crawled websites, periodic snapshots, open-source datasets, spam filters, phishing filters, and code-generation models as examples.
- What you can do after
- Evaluate how seemingly normal model behavior can conceal targeted failures caused by contaminated training data.
Read how Introduction to Generative AI, Second Edition explains poisoning attack →
The Art of AI Product Development
An attack that deliberately corrupts or manipulates data used to train or operate a machine-learning system in order to compromise its behavior or outputs.
- How it is used
- Frames poisoning attacks as an AI governance and data-security risk that can undermine system integrity and customer trust.
- How deep it goes
- Explains the attack, shows its product consequences, and gives validation, filtering, sourcing, and monitoring practices without detailing attack mechanics.
- Examples
- Uses DataMax recommendations, irrelevant industry advice, poisoned examples, production anomalies, and training-data validation as concrete illustrations.
- What you can do after
- Diagnose potential poisoning through anomalous data and output changes, then apply validation, filtering, trusted sourcing, and monitoring.
Read how The Art of AI Product Development explains poisoning attack →
LLMs in Production
Data poisoning is an attack that inserts malicious or manipulated examples into training data to alter a model's behavior.
- How it is used
- Frames training-data integrity as a security concern for large language model services.
- How deep it goes
- Defines the attack, illustrates its effects, and gives approximate image counts, but does not detail mitigation procedures.
- Examples
- Uses poisoned images, Nightshade, Midjourney, Stable Diffusion, and a code-word-triggered security bypass.
- What you can do after
- Evaluate training-data security by recognizing manipulated examples and behavior changes caused by them.
Related concepts
- training datasetuses · A collection of examples or observations supplied to a learning procedure to fit or adapt a model or learn its patterns or parameters.
- validationaddressed by · The process of checking data, behavior, systems, models, or results against defined criteria to assess their correctness, suitability, or acceptability.
- backdoor attackproduces · A backdoor attack uses or introduces a concealed mechanism that bypasses intended controls to enable unauthorized access or trigger behavior under specific conditions.
- adversarial inputimplements · An input crafted, selected, or modified to exploit a system’s weaknesses or induce incorrect, unsafe, or otherwise unintended behavior.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.