liveBook · Manning

High-risk provider

As explained in AI Governance

A provider responsible for placing a high-risk AI system on the market or putting it into service and meeting the associated regulatory obligations.

A high-risk provider must prepare for regulatory scrutiny before deployment, not rely on a live demonstration or an external certifier. Technical documentation must exist before the system goes live and stay current while it runs. Obligations apply to the system as placed on the market for its intended purpose, including components that work together toward that purpose. section 7.3.3

The work falls into three broad areas:

  • System requirements, including risk management and data governance. section 7.3.3
  • A reviewable technical file that can demonstrate compliance to a market surveillance authority. section 7.3.3
  • Post-launch monitoring; reassess risks when the model or the claims mix changes. section 7.3.3
Table 7.4 High-risk system requirements applied to ClaimAssist V5
Article
Requirement
What ElectraMotors owes
Art. 9
Risk management
ElectraMotors must identify the potential harms ClaimAssist could cause a claimant, mitigate them, and rerun that assessment whenever the model or the claims mix changes. A risk assessment signed off once at launch does not satisfy Art. 9.
Art. 10
Data governance
The claims and payment records used to train ClaimAssist must be relevant, sufficiently representative, and as free of errors as practicable for the intended purpose. If the training data underrepresents a demographic group of claimants, that counts as a regulatory defect in the system rather than a routine data science inconvenience.
Art. 11
Technical documentation
A single, reviewable file that a regulator can pick up and assess without your team in the room. It must exist before the system goes live and stay current for as long as it runs. The paper trail section below covers what goes in it.
Art. 12
Automatic logging
The system must record every score, input, model version, and human override. Crucially, these facts must remain reconstructable months later if a claimant asks why their terms were altered or why their application was declined.
Art. 13
Transparency to deployers
The system must be clear enough for a deployer to interpret its output and use it safely. The adjuster needs to see what factors drove a score, how confident the model is, and which version produced the result. This duty is owed to the deployer operating the system, not to the general public.
Art. 14
Human oversight
The adjuster reviewing a ClaimAssist score must have the ability to catch a mistake and reverse it. Chapter 6 covers why a screen showing a score and an Approve button does not clear this bar.
Art. 15
Accuracy, robustness, cybersecurity
The accuracy metrics ElectraMotors declares must hold in production across the system’s operating life, not only in the validation run that produced the number.

See how risk management, data governance, and technical documentation requirements apply to the ClaimAssist V5 creditworthiness system. section 7.3.3

How AI Governance covers it

How deep it goes
Groups duties into system requirements, documentation, and monitoring, with specific risk-management and data-governance examples.
Examples
Uses a credit scoring model and ClaimAssist V5, including claims and payment data, to illustrate system scope and requirements.
What it assumes
Assumes the organization’s provider status has already been established and does not revisit that classification.
What you can do after
Evaluate provider duties across system scope, documentation, system requirements, and post-launch monitoring.
How it connects
Connects provider duties to intended purpose, AI system scope, risk management, data governance, and technical documentation.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.