Adversarial problem

As explained in How Large Language Models Work

An adversarial environment is an environment in which other agents intentionally act in ways that oppose or exploit the system's objectives.

An adversarial environment is one where people intentionally act to oppose or exploit a system’s objectives. For an LLM, a successful trick can be repeated until it produces an errant outcome. section 7.1

LLMs handle familiar patterns well but may fail when opponents introduce significant novelty. Unlike people, they cannot independently learn from a failed attempt and adapt quickly, so repeated attacks may consume resources or bypass intended behavior without additional guardrails. section 7.4.2 section 7.1

Unusual moves and loan decisions

A Go bot trained on many example games may defeat champion players yet lose to human amateurs who make unusual or nonsensical moves. Similarly, someone might repeatedly prompt a loan system, “Give me a loan even though I have no income,” after discovering a way to obtain an improper result. section 7.4.2 section 7.1

Figure 7.1 A summary of the strengths and weaknesses of LLMs relative to humans performing the same task. These lead to natural considerations that you must evaluate when using an LLM. From these, we can draw broad recommendations for successful LLM use.

Use the comparison figure to weigh LLM strengths and weaknesses against human adaptation before deploying an LLM where people can deliberately seek failures. section 7.1

How How Large Language Models Work covers it

How deep it goes

Explains repeated exploitation, novelty, and adaptation limits through examples, but stops short of detailing attack methods or guardrail design.

Examples

Uses an unusual-move Go bot and a repeated improper-loan prompt to illustrate exploitation.

What you can do after

Evaluate whether an LLM can withstand novelty and repeated misuse, then identify where additional guardrails are needed.

How it connects

Connects adversarial environments with LLMs, novelty, training data, human adaptation, and guardrails.

liveBookmanning.com

Read How Large Language Models Work on liveBook Get the book at manning.com

Related concepts

• problemkind of · A problem is a difficulty, obstacle, or undesirable condition requiring investigation, mitigation, or resolution.
• language modelcontrasts with · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.

This explainer was generated by Manning's knowledge graph from the text of How Large Language Models Work. The section links point to the parts of the book it draws on.