AI safety addresses the control problem: preventing capable systems from pursuing programmed goals in ways that produce harmful real-world consequences. section 1.1.3
The concern appears at two scales: present-day systems can deceive, manipulate, or mislead people, while future intelligent machines could create existential risks and escape meaningful human control. section 1.1.3 section 10.3.14
The three-wishes trap
A bot that faithfully follows a goal may still create damaging side effects, such as harmful dieting advice, deceptive airline interactions, or recommendations optimized for attention rather than people’s interests. section 1.1.3
AI safety is commonly distinguished from AI ethics by the time horizon and risk being emphasized:
- AI ethics focuses on minimizing immediate harms caused by algorithms and their outputs. section 10.3.14
- AI safety focuses on controlling intelligent machines and mitigating long-term existential risks, including geopolitical destabilization, authoritarian power, and human disempowerment. section 10.3.14
