Adversarial input
An input crafted, selected, or modified to exploit a system’s weaknesses or induce incorrect, unsafe, or otherwise unintended behavior.
LLMs in Production
The study and use of inputs or models designed to manipulate, evade, or expose weaknesses in machine-learning systems.
- How it is used
- Frames security risks from inputs crafted to change model responses or bypass LLM instructions.
- How deep it goes
- Defines adversarial attacks and sketches prompt probing and data poisoning, without a detailed defense procedure.
- Examples
- Uses altered image pixels, repeated prompt queries, and attempts to extract secrets from an LLM.
- What you can do after
- Diagnose/debug with it when small input changes cause surprising outputs or prompts appear to bypass instructions.
Sutskever's List
An adversarial example is an input deliberately perturbed in a way that is often imperceptible to humans but causes a model to make an incorrect prediction.
- How it is used
- Uses adversarial examples to challenge claims that benchmark success reflects human-like understanding or general robustness.
- How deep it goes
- Defines the failure and its significance, but does not explain how perturbations are constructed.
- Examples
- Uses imperceptible input perturbations and a separate Google Photos mislabeling incident to illustrate brittleness.
- What you can do after
- Evaluate why benchmark performance alone does not establish robust visual understanding.
Generative AI in Action
An adversarial example is an input modified or crafted to cause a machine-learning model to produce an incorrect or undesirable output.
- How it is used
- Tests LLM resilience against harmful, bad, and edge-case inputs during enterprise red-teaming.
- How deep it goes
- Defines adversarial attacks and names generation tools, but does not explain attack algorithms or mathematical internals.
- Examples
- Uses TextAttack and a SLURM pipeline to run attacks and evaluate completions across Llama 2, Baichuan2, Mistral, and other models.
- What you can do after
- Apply it via TextAttack and attack pipelines to evaluate model resilience against problematic inputs.
Read how Generative AI in Action explains adversarial input →
Deep Learning with R, Third Edition
An adversarial example is an input deliberately or naturally modified in a small way that causes a machine-learning model to produce an incorrect output.
- How it is used
- Diagnoses brittleness in image models and language models by showing how minor input changes can cause major errors.
- How deep it goes
- Explains the idea through gradient ascent and examples, but does not develop attack algorithms or defenses.
- Examples
- Uses a panda-to-gibbon image change and an Alice in Wonderland riddle with altered numerical values.
- What you can do after
- Evaluate how apparently harmless input changes can expose mismatches between model predictions and human judgment.
Read how Deep Learning with R, Third Edition explains adversarial input →
Deep Learning with Python, Third Edition
Inputs intentionally or effectively perturbed to cause a model to make a wrong prediction despite minimal change to the original example.
- How it is used
- Exposes the brittleness of deep learning models and the mismatch between their input-output mappings and human perception.
- How deep it goes
- Explains the gradient-ascent mechanism and a panda-to-gibbon example, without detailing attack algorithms or defenses.
- Examples
- Uses a panda image modified toward the gibbon class and a figure illustrating imperceptible classification-changing perturbations.
- What you can do after
- Evaluate/understand how small input perturbations can cause confident model misclassification.
Read how Deep Learning with Python, Third Edition explains adversarial input →
Related concepts
- red-teamingused by · Red-teaming is an adversarial assessment practice that deliberately simulates attacks or uses challenging inputs to identify a system’s vulnerabilities, failures, or unsafe behavior.
- TextAttackused by · TextAttack is an open-source Python framework for generating adversarial examples, augmenting data, training NLP models, and evaluating their robustness.
- poisoning attackimplemented by · A poisoning attack deliberately corrupts data, a learning process, or model artifacts to compromise a machine-learning system’s behavior or outputs.
- robustnessmeasures · Robustness is the ability of a system or model to maintain acceptable behavior or performance despite variations, disturbances, faults, or challenging conditions.
- prompt injectionincludes · Prompt injection is an attack that introduces instructions through input or context to manipulate an AI system’s behavior or override its intended instructions.
- predicted valueproduces · A predicted value is an estimated outcome or output that a model produces for given input values.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.