liveBook · Manning

Serious incident

As explained in AI Governance

A serious incident is an event that directly or indirectly causes death, serious health damage, specified critical-infrastructure disruption, a breach of fundamental-rights obligations, or serious damage to property or the environment.

A single AI decision can appear to be corrected by human review while a harmful pattern persists across many users. Under the EU AI Act, a serious incident is an event that directly or indirectly causes death, serious health damage, serious and irreversible disruption of critical infrastructure management, a breach of fundamental-rights obligations, or serious damage to property or the environment. The threshold is high. section 7.5.2

A pattern missed by human oversight

A claims system flags applicants with a chronic condition as high risk, although the condition does not predict repayment. An adjuster corrects one recommendation, but other adjusters defer to the flag over several months. Because the condition is more prevalent in one ethnic group, the pattern may breach non-discrimination obligations and qualify as a serious incident. section 7.5.2

Turn detection into a response workflow:

  • Monitor system performance and investigate significant changes, including differences across demographic groups or applicant segments. section 7.3.1
  • Assess whether the harm meets the serious-incident threshold. The reporting clock starts when a causal link is reasonably likely, not when the root cause is confirmed. section 7.5.2
  • A deployer notifies both the provider and the market surveillance authority. The provider must file its own formal report, even if the deployer detected the incident first. section 7.3.1 section 7.3.3
  • Investigate and correct the failure without delay; restrict, withdraw, recall, or shut down the system when necessary. Prepare rollback and shutdown capabilities before they are needed. section 7.3.3
Figure 7.14 EU AI Act incident reporting tiers. Clock starts at reasonable likelihood of a causal link, not confirmed root cause.

Use the reporting tiers to navigate the applicable reporting clock; do not wait for a confirmed root cause. section 7.5.2

How AI Governance covers it

How deep it goes
Defines qualifying harms and outlines detection, reporting, and corrective action without specifying deadlines in the available evidence.
Examples
Uses biased claims scoring and autonomous vehicles failing to stop for school buses as potential serious-incident scenarios.
What you can do after
Evaluate potential serious incidents and identify provider and deployer reporting responsibilities.
How it connects
Connects incident reporting to fundamental rights, human oversight, performance monitoring, and corrective action.

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.