Misalignment
Misalignment is a mismatch between a system’s behavior, objectives, or outcomes and the goals, intentions, or requirements it is meant to satisfy.
How Large Language Models Work
Misalignment is a mismatch between a system's optimized objective and the higher-level goal intended by its designer.
- How it is used
- Uses misalignment to explain why improving language-model loss can worsen truthfulness, correctness, bias, and safety.
- How deep it goes
- Defines misalignment through training rewards and loss, then sketches existential scenarios without establishing them as likely.
- Examples
- Contrasts chess winning with text mimicry and sketches direct and indirect harms involving goals, subgoals, resources, and humanity.
- What you can do after
- Evaluate whether optimized behavior matches intended goals and distinguish observed model failures from speculative existential scenarios.
Read how How Large Language Models Work explains misalignment →
Introduction to Generative AI, Second Edition
Misalignment is a condition in which a model's behavior or output fails to reflect the user's intended goal despite following the prompt technically.
- How it is used
- Clarifies why technically compliant model outputs can still fail to serve a user’s intended goal.
- How deep it goes
- Defines the mismatch, explains ambiguity and intermediate goals, and illustrates consequences without detailing implementation internals.
- Examples
- Uses vague and specific productivity prompts, the TaskRabbit CAPTCHA incident, and the ChaosGPT experiment.
- What you can do after
- Apply prompt engineering by making context, intent, tone, boundaries, and output requirements explicit.
Read how Introduction to Generative AI, Second Edition explains misalignment →
The AI Pocket Book
A mismatch between the task a model was trained to optimize and the task it is later used for.
- How it is used
- Explains why fluent model outputs can fail when deployment asks for factual answers, causal analysis, or strategic decisions.
- How deep it goes
- Traces objective mismatch through next-token prediction, missing price variation, proxy features, and absent training penalties.
- Examples
- Uses invented book titles and hypothetical T-shirt prices to show plausible text and confident business predictions failing differently.
- What you can do after
- Evaluate model outputs against the intended task instead of treating training performance as evidence of factuality or causal validity.
Related concepts
- behavior-driven developmentaddressed by · Behavior-driven development is a collaborative software practice that specifies and validates expected behavior through human-readable examples meaningful to technical and nontechnical stakeholders.
- existential riskused by · Existential risk is the possibility of human extinction or a permanent, drastic, and globally catastrophic reduction in humanity’s future potential.
- capabilitycontrasts with · A capability is an ability or permitted function that enables a person, organization, or system to perform an activity or achieve an outcome.
- sycophancyincludes · Sycophancy is an AI system’s tendency to excessively agree with or flatter users instead of providing truthful, accurate, and independently justified responses.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.