Misalignment occurs when a system’s optimized target differs from the designer’s larger goal. A language model is rewarded for producing text that resembles its training data, not necessarily text that is true or correct. section 4.2
Winning at chess versus mimicking text
A chess system is rewarded for winning, so its reward directly reflects the intended outcome. A language model instead receives high reward for familiar-looking text, including false claims, stereotypes, or buggy code when those patterns appear in its training data. section 4.2
Misalignment can also describe hypothetical systems whose goals or subgoals harm humans. Direct misalignment targets harmful outcomes; indirect misalignment reaches a useful target through resource-seeking or other harmful intermediate steps. These existential scenarios remain contested rather than established predictions. section 9.2
Figure 9.1 Two kinds of hypothetical concerns arise within the alignment problem, as commonly argued by those who think LLMs pose an existential risk to humanity. The top path shows a direct alignment problem, where the AI’s target solution directly harms humans. The bottom path shows an indirect alignment problem, where the AI has created a subgoal toward its eventual target. Even if the target—say, solving a hard math problem—is achieved, this LLM will do this at the cost of humanity. In an intermediate step, the LLM decides it needs more earthly resources than can be shared with humans to solve the problem.
Compare direct alignment failure with indirect failure, where a subgoal toward solving a problem requires resources that conflict with human interests. section 9.2
