Poisoning attack
A poisoning attack deliberately corrupts data, a learning process, or model artifacts to compromise a machine-learning system’s behavior or outputs.
Introduction to Generative AI, Second Edition
Data poisoning is an attack that inserts malicious or misleading examples into training data so a model learns harmful or exploitable behavior.
- How it is used
- Explains how attackers influence generative AI behavior by corrupting training or post-training data.
- How deep it goes
- Describes attack timing, dataset exposure, targeted failures, and broad consequences without detailing a complete defense procedure.
- Examples
- Uses crawled websites, periodic snapshots, open-source datasets, spam filters, phishing filters, and code-generation models as examples.
- What you can do after
- Evaluate how seemingly normal model behavior can conceal targeted failures caused by contaminated training data.
Read how Introduction to Generative AI, Second Edition explains poisoning attack →
The Art of AI Product Development
An attack that deliberately corrupts or manipulates data used to train or operate a machine-learning system in order to compromise its behavior or outputs.
- How it is used
- Frames poisoning attacks as an AI governance and data-security risk that can undermine system integrity and customer trust.
- How deep it goes
- Explains the attack, shows its product consequences, and gives validation, filtering, sourcing, and monitoring practices without detailing attack mechanics.
- Examples
- Uses DataMax recommendations, irrelevant industry advice, poisoned examples, production anomalies, and training-data validation as concrete illustrations.
- What you can do after
- Diagnose potential poisoning through anomalous data and output changes, then apply validation, filtering, trusted sourcing, and monitoring.
Read how The Art of AI Product Development explains poisoning attack →
LLMs in Production
Data poisoning is an attack that inserts malicious or manipulated examples into training data to alter a model's behavior.
- How it is used
- Frames training-data integrity as a security concern for large language model services.
- How deep it goes
- Defines the attack, illustrates its effects, and gives approximate image counts, but does not detail mitigation procedures.
- Examples
- Uses poisoned images, Nightshade, Midjourney, Stable Diffusion, and a code-word-triggered security bypass.
- What you can do after
- Evaluate training-data security by recognizing manipulated examples and behavior changes caused by them.
AI Governance
Model poisoning is the deliberate or accidental introduction of harmful, misleading, or biased data or updates that alter a model's behavior.
- How it is used
- Frames poisoning as a model-weight risk when securing generative AI systems.
- How deep it goes
- Defines poisoning and distinguishes it from fine-tuning degradation, but stops short of attack mechanics or mitigations.
- What you can do after
- Understand poisoning as a behavior-altering risk embedded in a model rather than external code executed during loading.
Also covered in
- Architecting for AutonomyIn early access (MEAP)
- Generative AI in Action
- Retrieval Augmented Generation, The Foundational IdeasIn early access (MEAP)
- Digital Twins in Action
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- sensitive informationSensitive information is information whose access, disclosure, alteration, or misuse may cause harm or violate privacy, security, or confidentiality requirements.
- contractA contract is an agreed specification of the conditions, expectations, rights, or obligations that govern behavior or interactions between parties or software components.
Related concepts
- training datasetuses · A training dataset is a collection of examples supplied to a learning procedure to fit or adapt a model’s parameters or behavior.
- adversarial inputimplements · An input crafted, selected, or modified to exploit a system’s weaknesses or induce incorrect, unsafe, or otherwise unintended behavior.
- backdoor attackproduces · A backdoor attack uses or introduces a concealed mechanism that bypasses intended controls to enable unauthorized access or trigger behavior under specific conditions.
- validationaddressed by · Validation is the process of checking data, behavior, or results against defined criteria to determine their correctness, acceptability, or suitability for a purpose.
Questions about poisoning attack
What is poisoning attack?
A poisoning attack deliberately corrupts data, a learning process, or model artifacts to compromise a machine-learning system’s behavior or outputs.
Which Manning books explain poisoning attack?
Introduction to Generative AI, Second Edition, The Art of AI Product Development, LLMs in Production and AI Governance.
What should I know before learning poisoning attack?
Manning books that teach poisoning attack usually introduce language model, sensitive information and contract first.
What is poisoning attack part of?
Poisoning attack is part of model weight risk. A model weight risk is a security or safety risk embedded in a model’s learned parameters rather than introduced by external code executed during model loading.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



