How deeply does LLMs in Production cover adversarial input?

Defines adversarial attacks and sketches prompt probing and data poisoning, without a detailed defense procedure.

Adversarial input

The study and use of inputs or models designed to manipulate, evade, or expose weaknesses in machine-learning systems.

A tiny change to an input can produce a large change in a model’s answer. An image may look unchanged to a person yet be misclassified after its pixels are altered; a slightly changed LLM prompt can likewise produce a different response. Adversarial inputs exploit this gap between what people perceive and how models respond. section 6.3.6

For LLMs, prompt injection is a practical concern: a user crafts an input to make the model ignore its instructions and produce unwanted content. The stakes extend beyond an odd answer when an application gives a model access to company data, secrets, code execution, or a file system. A model may also invent a plausible-looking secret, so an apparent disclosure is not proof that the secret was real. section 3.2.9

Adversarial pressure can reach a model in several ways:

An attacker can send many candidate prompts, collect responses, and use that dataset to predict which prompts are likely to get a desired output. section 6.3.6
With data poisoning, an attacker instead places malicious examples in training data to change later model behavior. section 6.3.6
A detector for AI-generated writing faces an adversarial loop: if its decisions can reliably distinguish generated text, a new model can be developed to evade it. section 12.2.1

Input restrictions involve trade-offs, not automatic fixes. Rejecting prompts outside supported languages sacrifices usability and requires safeguards for each supported language; a multilingual prompt may also bypass a detector that identifies only one language. Flagging unsupported-language prompts for closer monitoring is another option. section 6.3.6

How LLMs in Production covers it

liveBook manning.com

Read LLMs in Production on liveBook Get the book at manning.com

Also explained in

liveBook manning.com

Sutskever's ListAn adversarial example is an input deliberately perturbed in a way that is often imperceptible to humans but causes a model to make an incorrect prediction.
liveBook manning.com

Generative AI in ActionAn adversarial example is an input modified or crafted to cause a machine-learning model to produce an incorrect or undesirable output.
liveBook manning.com

Deep Learning with R, Third EditionAn adversarial example is an input deliberately or naturally modified in a small way that causes a machine-learning model to produce an incorrect output.
liveBook manning.com

Deep Learning with Python, Third EditionInputs intentionally or effectively perturbed to cause a model to make a wrong prediction despite minimal change to the original example.

Compare how 5 books explain adversarial input →

Related concepts

red-teamingused by · Red-teaming is an adversarial assessment practice that deliberately simulates attacks or uses challenging inputs to identify a system’s vulnerabilities, failures, or unsafe behavior.
TextAttackused by · TextAttack is an open-source Python framework for generating adversarial examples, augmenting data, training NLP models, and evaluating their robustness.
poisoning attackimplemented by · A poisoning attack deliberately corrupts data, a learning process, or model artifacts to compromise a machine-learning system’s behavior or outputs.
robustnessmeasures · Robustness is the ability of a system or model to maintain acceptable behavior or performance despite variations, disturbances, faults, or challenging conditions.
prompt injectionincludes · Prompt injection is an attack that introduces instructions through input or context to manipulate an AI system’s behavior or override its intended instructions.
predicted valueproduces · A predicted value is an estimated outcome or output that a model produces for given input values.

This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.