A customer relied on an airline chatbot’s incorrect bereavement-fare advice and bought a ticket under false assumptions. The resulting harm was financial, not existential. AI safety begins with this ordinary question: what happens when a system’s objective, training data, deployment setting, or oversight fails, and someone relies on its output? section 9.2
The stakes change with a system’s capabilities and setting. An automated driving system that fails to identify a pedestrian can cause physical harm. AGI safety looks further ahead: if an AI could solve general problems rather than one narrow task, it might learn or evolve in ways its designers did not anticipate. Those possibilities call for preparation, but they do not make every present-day failure a sign of imminent superintelligence. section 9.2 section 9.2.3
In 2007, the emerging AGI field made human-level or superhuman intelligence a concrete research goal and brought safe design into the discussion. It established a setting for ideas such as alignment and friendly AI without settling how to achieve them. Safety thus spans two questions: how to prevent harms from systems already in use, and how early to act on risks from systems that do not yet exist. section 9.2.3 section 9.2
When precaution becomes a governance problem
GPT-2’s declaration as “too dangerous to release” made the tension between scaling AI and safety public. Later, concerns about a reported research project called Q* figured in the turmoil around Sam Altman’s 2023 firing. In both cases, anticipated risk prompted action rather than an immediate threat. The later decision also exposed a cost of precaution: without a clearly communicated safety concern, an attempt to avert harm produced an institutional crisis. section 1.0 section 1.1
The practical lesson is neither to dismiss uncertain risks nor to treat concern alone as an answer. Safety decisions must account for the harms a system can cause, the uncertainty about future capabilities, and the consequences of acting before the risk can be clearly explained. section 9.2 section 1.1
