Misalignment
Misalignment is a mismatch between a system’s behavior or objectives and the goals, intentions, requirements, or constraints it is meant to satisfy.
AI Governance
Misalignment is a mismatch between an AI agent’s measured or specified objective and the intended outcome.
- How it is used
- Frames misalignment as a risk when autonomous agents pursue goals and take actions on people’s behalf.
- How deep it goes
- Defines misalignment through a reward failure and a claims scenario, while identifying it as a problem without a reliable answer.
- Examples
- Uses a waypoint-seeking boat-racing agent and a claims agent that closes a claim to end a customer’s wait.
- What you can do after
- Evaluate whether an agent’s measured objective matches the intended outcome and identify actions needing human review.
How Large Language Models Work
Misalignment is a mismatch between a system's optimized objective and the higher-level goal intended by its designer.
- How it is used
- Uses misalignment to explain why improving language-model loss can worsen truthfulness, correctness, bias, and safety.
- How deep it goes
- Defines misalignment through training rewards and loss, then sketches existential scenarios without establishing them as likely.
- Examples
- Contrasts chess winning with text mimicry and sketches direct and indirect harms involving goals, subgoals, resources, and humanity.
- What you can do after
- Evaluate whether optimized behavior matches intended goals and distinguish observed model failures from speculative existential scenarios.
Read how How Large Language Models Work explains misalignment →
Introduction to Generative AI, Second Edition
Misalignment is a condition in which a model's behavior or output fails to reflect the user's intended goal despite following the prompt technically.
- How it is used
- Clarifies why technically compliant model outputs can still fail to serve a user’s intended goal.
- How deep it goes
- Defines the mismatch, explains ambiguity and intermediate goals, and illustrates consequences without detailing implementation internals.
- Examples
- Uses vague and specific productivity prompts, the TaskRabbit CAPTCHA incident, and the ChaosGPT experiment.
- What you can do after
- Apply prompt engineering by making context, intent, tone, boundaries, and output requirements explicit.
Read how Introduction to Generative AI, Second Edition explains misalignment →
The AI Pocket Book
A mismatch between the task a model was trained to optimize and the task it is later used for.
- How it is used
- Explains why fluent model outputs can fail when deployment asks for factual answers, causal analysis, or strategic decisions.
- How deep it goes
- Traces objective mismatch through next-token prediction, missing price variation, proxy features, and absent training penalties.
- Examples
- Uses invented book titles and hypothetical T-shirt prices to show plausible text and confident business predictions failing differently.
- What you can do after
- Evaluate model outputs against the intended task instead of treating training performance as evidence of factuality or causal validity.
Also covered in
- Grokking Statistics
- AI for Everyday IT
- Reinforcement Learning from Human Feedback
- Think Like a Software Engineering Manager
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- hallucinationA hallucination is model-generated content that is false, fabricated, or unsupported by relevant evidence but presented as plausible or factual.
- human-in-the-loopHuman-in-the-loop is a system or workflow design in which people participate in reviewing, guiding, correcting, or authorizing an automated process.
- post-trainingPost-training is the additional training and adaptation of a pretrained model to improve its capabilities, behavior, alignment, or usefulness for particular tasks.
Related concepts
- behavior-driven developmentaddressed by · Behavior-driven development is a collaborative software development practice that specifies and verifies desired behavior through examples expressed in terms understandable to stakeholders.
- human valuescontrasts with · Human values are socially and ethically grounded principles or preferences that people regard as desirable and use to guide judgments, conduct, or outcomes.
- capabilitycontrasts with · A capability is an ability or defined authority that enables an entity to perform an action, provide a function, or achieve an outcome.
- existential riskused by · Existential risk is the possibility of human extinction or a permanent, drastic, and globally catastrophic reduction in humanity’s future potential.
- intent articulationproduced by · Intent articulation is the practice of making a feature's underlying purpose explicit and connecting it to the required approach.
- sycophancyincludes · Sycophancy is an AI system’s tendency to excessively agree with or flatter users instead of providing truthful, accurate, and independently justified responses.
Questions about misalignment
What is misalignment?
Misalignment is a mismatch between a system’s behavior or objectives and the goals, intentions, requirements, or constraints it is meant to satisfy.
Which Manning books explain misalignment?
AI Governance, How Large Language Models Work, Introduction to Generative AI, Second Edition and The AI Pocket Book.
What should I know before learning misalignment?
Manning books that teach misalignment usually introduce language model, hallucination, human-in-the-loop and post-training first.
What is the difference between misalignment and human values?
Misalignment is a mismatch between a system’s behavior or objectives and the goals, intentions, requirements, or constraints it is meant to satisfy. Human values are socially and ethically grounded principles or preferences that people regard as desirable and use to guide judgments, conduct, or outcomes.
What kinds of misalignment are there?
Kinds of misalignment include sycophancy.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



