Chapter 7: Regulation and Compliance
The introduction explains why teams need to clarify their EU AI Act obligations and how prior governance work can prepare them for discussions with Legal.
After this chapter you will be able to
- Uncertain AI risk classifications. Explain why a team should not assume its AI system falls outside the EU AI Act’s high-risk category: enterprise systems can be difficult to classify, and relevant obligations have begun taking effect.
- Governance evidence and legal judgment. Explain how the governance practices recommended in Chapters 3 through 6 relate to compliance preparation, and distinguish that preparation from legal advice.
- AI Act Reach. Explain that the EU AI Act can apply when a system’s output is used in the EU, regardless of where its company is based, and that internal systems can also be covered.
- Risk Tier Distinction. Describe the section’s basic distinction between limited-risk human-facing or content-generating systems and minimal-risk background systems.
- Provider and Deployer Roles. Distinguish the basic provider and deployer roles and state which carries heavier obligations.
- Intended Purpose and Risk. Explain why classification depends on a system’s intended purpose and context of use rather than its underlying technology or broad industry. Describe why ClaimAssist changed from limited risk to high risk.
- Classification Screening Questions. State the classification questions presented here in order: whether it is an AI system, whether the use is prohibited, whether it is high risk, and what transparency rules apply to everything else.
- Inference as the Main Test. Explain why inference is the main separator between an AI system and software that executes explicitly written rules. Describe the other elements of the definition and the kinds of outputs that may bring a system into scope.
- RAG Pipeline Inference. Explain why a retrieval-augmented generation pipeline is in scope despite retrieving documents rather than training a model.
- Agent Toolset Risk Boundary. Explain why an agent and its tools are judged as one system and how adding a tool can change the application’s regulatory risk profile.
- Distinct Components and Documentation. Describe when linked components need separate classifications, when a routing step usually does not, and how to document an ambiguous decision.
- Outright Prohibition and Workplace Emotions. Explain that an Article 5 prohibition stops deployment, regardless of the vendor’s label. Describe why a tool that infers employees’ emotions from facial expressions or voice patterns is prohibited unless used for a medical or safety reason.
- Social Scoring Boundary. Describe when a behavior-based score becomes prohibited social scoring, using the home-sharing example to distinguish general character scoring from checks based on specific, relevant, verifiable signals.
- Sensitive Biometric Categorization. Distinguish prohibited sorting of people by sensitive attributes from permitted demographic auditing, and explain why an apparently unlisted attribute can still pose a proxy risk.
- Manipulation and Vulnerability Thresholds. Distinguish the technique-based manipulation ban from the ban on exploiting vulnerable targets, and describe the harm and decision-making thresholds the section attaches to them.
- Early Documented Prohibition Checks. Describe when to run a prohibition check and what to record if a proposed system appears close to, but outside, a ban.
- Purpose Based Risk Classification. Explain why the same model can receive different risk classifications in different deployments, and describe how an intended use in an Annex III domain can make a system high-risk.
- Self Assessment and Profiling Limits. Describe when a provider may justify treating an Annex III system as not high-risk, why genuine independent human review matters, and how profiling limits that option.
- Consequential Use Gray Zones. Apply the section’s three questions to distinguish a policy explanation from a potentially high-risk recommendation, and describe what to record when the answer is uncertain.
- Agent Classification and Drift. Explain how agents are classified and why their classification must be revisited as their tools and actions change.
- Model and system obligations. Distinguish a GPAI model provider’s compliance duties from the duties of an organization building a system on that model.
- Downstream documentation evidence. Describe the provider information needed for a downstream compliance case and how to obtain and maintain it, whether the model is integrated directly or sits behind another vendor’s application.
- GPAI provider boundary. Distinguish ordinary downstream customization from offering a modified model to others as a general-purpose model.
- AI Interaction Disclosure. Explain when a person must be told they are interacting with AI, who owns the disclosure, and how the narrow obvious-context exception works.
- Generated Media Metadata. Describe the machine-readable marking required for AI-generated images, audio, and video, including how responsibility changes when an organization builds its own generation tool.
- Visible Deepfake Labels. Explain when published deepfakes need a human-readable label and how that duty differs from embedded media marking.
- Role-Specific AI Literacy. Explain the AI literacy duty for organizations using AI and why training should reflect staff roles and the AI tools actually in use.
- ClaimAssist Risk Threshold. Explain why the FAQ version of ClaimAssist has transparency duties without being high-risk, and what change would cross the high-risk threshold.
- Responsibility-Based Provider Status. Explain why the financial services client is the provider of the creditworthiness system despite the contract calling it a deployer, and how that left it exposed when required documentation was missing.
- Provider status and risk tiers. Explain when an organization is a provider, including when it puts an internal system into use, and how the provider’s obligations differ by risk tier.
- Application ownership versus configuration. Distinguish building a specific-purpose system from a model API or vendor components from configuring a vendor’s product within its intended purpose.
- Deployer oversight and system fit. Describe a deployer’s operational duties and why it must check that a system is used within the provider’s boundaries.
- Branding a high-risk system. Explain why putting your name or trademark on an existing high-risk AI system and making it available to others makes you its provider.
- Changing the intended purpose. Explain how using a system for a new purpose that makes it high-risk can turn its deployer into its provider.
- Substantially modifying the system. Describe how a substantial modification can make the organization the provider of a modified AI system, using the bank’s automation example.
- Scope of the regulated system. Explain why high-risk obligations apply to the system as provided for its intended purpose, not just to its predictive model. Describe the components included in ClaimAssist V5.
- Internal assessment and current evidence. Explain why providers of credit models and most Annex III systems must prepare official technical documentation before launch and keep it current.
- Reviewable documents and deployer instructions. Distinguish the technical documentation package from the instructions for use, and describe what each must enable its reader to do.
- Monitoring and response duties. Describe how a provider must monitor a high-risk system after launch and respond when monitoring identifies a problem or a serious incident.
- Separate deployer duties and contracts. Explain why provider compliance does not cover the deployer’s duties, and describe the documentation and contractual access a deployer should secure before signing.
- Validated inputs and populations. Explain why a provider’s accuracy and fairness claims may not hold for a deployer’s chosen inputs or population, and state the options when deployment falls outside the validated context.
- Effective human oversight. Describe what designated overseers of a high-risk system must be able and authorized to do, and distinguish that from routinely clicking through outputs.
- Notices and individual responses. Describe whom a high-risk deployer must inform and the two kinds of individual response it should prepare before launch.
- Operational monitoring and response. Describe what a deployer should do when monitoring reveals reasonable grounds for risk, alongside its log-retention and authority-cooperation duties.
- Supply Chain Documentation Rights. Distinguish the documentation owed to a system provider integrating a GPAI model from the instructions owed to a deployer of a finished high-risk product. Explain how to check delivery against the GPAI Code of Practice and whom to contact when a provider is outside the EU.
- Open Source Documentation Limits. Explain what an open-source GPAI provider must still disclose, what documentation it need not provide on request, and why a downstream high-risk user may have to fill the remaining gap.
- Layered Vendor Contract Boundaries. Explain why a customer of a SaaS product must seek documentation and data protections through its direct vendor, rather than assuming it can enforce terms against the underlying model provider.
- Model Evidence and Version Terms. Describe the technical information and model-version commitments a supplier contract should secure, including why the update-notice window must fit re-validation.
- Logs and Compliance Allocation. Explain how a supplier contract should secure usable logs, allocate compliance work, and require notice of serious incident reports.
- Fine-Tuning and Provider Status. Distinguish fine-tuning a model from the acts that can make an organization a provider, including releasing a heavily fine-tuned GPAI model or substantially modifying a deployed high-risk system.
- New Purpose Through RAG. Explain why adding retrieval to a model API can still make the organization the provider of a new AI system when it gives the model a specific high-stakes purpose.
- Agent Platform Role Ownership. Explain how a team using an agent-building platform can be the provider of the resulting agent system despite relying on a turnkey vendor platform.
- Combined Classification Review. Describe how and when a team should assess both a system's risk classification and its provider or deployer role, and where it should record the result.
- Management Certification Versus AI Act Compliance. Explain why an ISO 42001 certificate does not establish AI Act compliance, and distinguish the additional requirements the section identifies for providers and deployers.
- Framework Head Start and Limits. Explain why ISO 42001 or the NIST AI RMF helps an organization prepare for AI Act compliance without being sufficient on its own.
- Remaining High-Risk Provider Duties. Describe the provider obligations ISO 42001 does not supply, distinguishing formal compliance steps from the AI Act’s more specific quality-management and deployer-documentation requirements.
- Uncovered High-Risk Deployer Obligations. State the deployer-specific AI Act obligations that ISO 42001 does not cover and explain that its coverage of deployer activity is weaker than its provider coverage.
- Reviewable provider evidence pack. Explain why a provider needs a coherent evidence pack rather than a vendor contract, certification, or scattered tickets, and why it must be assembled during development and retained after launch.
- High-risk conformity assessment routes. Describe the usual conformity assessment for a high-risk system and the two exceptions described here.
- Technical and legal evidence ownership. Distinguish the technical team’s evidence responsibilities from the lawyer’s regulatory responsibilities, and describe how a single owned repository supports their handoff.
- Documented joint risk classification. Explain why high-risk classification requires both technical and legal input and what the lasting classification record should contain.
- Deployer Evidence Responsibility. Distinguish a deployer’s evidence trail from a builder’s: a deployer must show that it selected the right system, configured it for its intended use, and operated it as the provider designed.
- Proof of Actual Operation. Distinguish written oversight and monitoring plans from evidence that those plans have been carried out.
- Compliance After System Changes. Explain how a post-launch model update can make an earlier risk assessment and evidence pack outdated, even after a system has completed its initial compliance steps.
- Ongoing Provider and Deployer Duties. Distinguish the post-deployment duties stated for providers and deployers, and describe when a significant system change may call for reassessment.
- Provider monitoring plan. Describe the provider's post-market monitoring plan and distinguish it from the QMS.
- Deployer monitoring responsibilities. Distinguish the deployer's monitoring work from the provider's monitoring-plan responsibility.
- Drift and reassessment triggers. Explain what monitoring is meant to catch, when reassessment is warranted, and how documented expected changes affect re-filing.
- Personal data in logs. Explain how to reconcile monitoring logs with privacy requirements, including deletion requests and free-text prompt logs.
- Serious incident threshold. Explain what makes an AI failure a potentially serious incident and apply the section’s harm, pattern, and persistence questions to ClaimAssist.
- Reporting tiers and clock. Describe the three AI Act reporting deadlines and when their clocks begin.
- Deployer and provider duties. Distinguish the deployer’s immediate notification duty from the provider’s formal reporting and corrective-action duties.
- Sandbox benefits and limits. Explain how a regulatory sandbox lets providers and deployers test AI systems under supervision before launch, and why its results can strengthen a later evidence pack without replacing conformity assessment. Describe why an organization should not wait for a sandbox if none is available.
- Smaller organization relief. Distinguish the AI Act’s penalty calculation for SMEs and startups from that for larger companies, and explain the proposed small mid-cap relief. State why smaller organizations still owe the same substantive high-risk compliance work.
- Provider Role and Risk Classification. Distinguish ElectraMotors’ provider role from ClaimAssist’s risk classification, and explain what changed at V5.
- Traceable Governance Evidence. Explain how earlier governance work affects the effort of assembling an AI Act evidence pack.
What it builds on
- The Six Level Governance Framework (chapter 3) · The governance framework introduced earlier in the book.
- Operations & Monitoring (Level 5) (chapter 3) · Operational monitoring provides the context for observing changes in system use.
- Risk & Impact Assessment (Level 2) (chapter 3) · The AI Impact Assessment is where the classification and reasoning are recorded.
- What Does “Agentic” Mean? (chapter 2) · The distinction between an agent and a conventional AI application.
- Design & Implementation Reviews (Level 3) (chapter 3) · The design and implementation stage before deployment.
- Overreliance (chapter 6) · Overreliance helps explain why nominal human review may cease to be independent.
- Common properties of Agentic AI (chapter 2) · Agents can use tools and carry out actions beyond producing a static response.
- Key artifacts (chapter 3) · The organizational AI inventory used to identify tools in use.
Section by section
Introduction
The introduction explains why teams need to clarify their EU AI Act obligations and how prior governance work can prepare them for discussions with Legal.
- Uncertain AI risk classifications: Explain why a team should not assume its AI system falls outside the EU AI Act’s high-risk category: enterprise systems can be difficult to classify, and relevant obligations have begun taking effect.
Why it matters: Classification uncertainty is a reason to investigate obligations rather than assume they do not apply. - Governance evidence and legal judgment: Explain how the governance practices recommended in Chapters 3 through 6 relate to compliance preparation, and distinguish that preparation from legal advice.
Why it matters: Existing governance work can support compliance without replacing legal judgment.
Key terms: high-risk, risk tier, legal obligations, evidence, regulators, legal advice, Legal
7.1 The Regulatory Landscape
Introduces the EU AI Act as a reference for AI compliance, explaining its reach, basic risk and role distinctions, and the need to check other applicable laws.
- AI Act Reach: Explain that the EU AI Act can apply when a system’s output is used in the EU, regardless of where its company is based, and that internal systems can also be covered.
Why it matters: A team cannot rule out the Act just because it is based outside the EU or uses AI only internally. - Risk Tier Distinction: Describe the section’s basic distinction between limited-risk human-facing or content-generating systems and minimal-risk background systems.
Why it matters: Even an internal tool may be above minimal risk if people interact with it or it generates content. - Provider and Deployer Roles: Distinguish the basic provider and deployer roles and state which carries heavier obligations.
Why it matters: A system’s risk tier alone does not tell a team the weight of its obligations. - EU Baseline and Other Laws: Explain why the chapter uses the EU AI Act as its primary reference while still requiring checks for other applicable laws and local guidance.
Why it matters: Using one strong baseline simplifies planning, but it does not replace checking every law that applies to a deployment.
Key terms: EU AI Act, output, internal systems, risk tiers, limited-risk, minimal-risk, human-facing output, provider, deployer, obligations
7.2 Classifying Your System
This section explains how an AI system’s intended purpose determines its risk classification, why that classification may need to change, and how to document the decision.
- Intended Purpose and Risk: Explain why classification depends on a system’s intended purpose and context of use rather than its underlying technology or broad industry. Describe why ClaimAssist changed from limited risk to high risk.
Why it matters: A change in what a product does can change its regulatory classification even if it remains the same product. - Actual Use and Reclassification: Describe how observed use outside the approved intended purpose can signal that a system needs to be reclassified.
Why it matters: Watching how a system is actually used helps catch classification changes that product paperwork has not yet reflected. - Classification Screening Questions: State the classification questions presented here in order: whether it is an AI system, whether the use is prohibited, whether it is high risk, and what transparency rules apply to everything else.
Why it matters: The questions provide an initial route through classification without assuming every AI use is high risk. - Documented Classification Reasoning: Explain what teams should rely on while final classification guidelines are pending and how they should handle and record a gray-zone decision.
Why it matters: A written rationale lets the team explain its classification if a regulator questions it.
Key terms: intended purpose, limited risk, high-risk, Annex III, actual use, approved intended purpose, classification change, AI system, prohibited, transparency rules
7.2.1 Is It an AI System at All?
This section explains how to decide whether an application or component is an AI system, with particular attention to inference and the boundaries of assembled systems.
- Inference as the Main Test: Explain why inference is the main separator between an AI system and software that executes explicitly written rules. Describe the other elements of the definition and the kinds of outputs that may bring a system into scope.
Why it matters: This test helps distinguish AI systems from ordinary automated software before assessing their regulatory obligations. - Assembled Application Boundary: Distinguish an operating AI application from idle model weights or a static artifact it produced. Explain why an embedded module can qualify even when users do not see it.
Why it matters: Choosing the right boundary prevents teams from overlooking embedded AI or classifying a static output as a system. - RAG Pipeline Inference: Explain why a retrieval-augmented generation pipeline is in scope despite retrieving documents rather than training a model.
Why it matters: Calling RAG “just retrieval” can cause a team to miss an AI system it needs to classify. - Agent Toolset Risk Boundary: Explain why an agent and its tools are judged as one system and how adding a tool can change the application’s regulatory risk profile.
Why it matters: A team must reconsider classification when it gives an agent consequential new capabilities. - Distinct Components and Documentation: Describe when linked components need separate classifications, when a routing step usually does not, and how to document an ambiguous decision.
Why it matters: Component-level reasoning and a documented basis make classification decisions defensible.
Key terms: inference, AI system, explicit rules, fully assembled application, model weights, Retrieval-augmented generation (RAG), database lookup, agent, toolset, risk profile
7.2.2 Prohibited practices
This section explains how to recognize several AI Act prohibited practices, distinguish them from nearby lawful uses, and check for prohibitions before deployment.
- Outright Prohibition and Workplace Emotions: Explain that an Article 5 prohibition stops deployment, regardless of the vendor’s label. Describe why a tool that infers employees’ emotions from facial expressions or voice patterns is prohibited unless used for a medical or safety reason.
Why it matters: A prohibited use cannot be made launch-ready by treating it as an ordinary compliance issue. - Social Scoring Boundary: Describe when a behavior-based score becomes prohibited social scoring, using the home-sharing example to distinguish general character scoring from checks based on specific, relevant, verifiable signals.
Why it matters: It helps distinguish a relevant service check from an exclusionary judgment about someone’s general character. - Sensitive Biometric Categorization: Distinguish prohibited sorting of people by sensitive attributes from permitted demographic auditing, and explain why an apparently unlisted attribute can still pose a proxy risk.
Why it matters: The purpose of categorization and the dataset’s proxy relationships both affect the prohibition check. - Manipulation and Vulnerability Thresholds: Distinguish the technique-based manipulation ban from the ban on exploiting vulnerable targets, and describe the harm and decision-making thresholds the section attaches to them.
Why it matters: Checking only the tactic or only the target can miss a prohibited use. - AI Act and Dark Patterns: Explain how AI-driven manipulation can overlap with the Digital Services Act’s dark-pattern prohibition without assuming the two laws have identical scope.
Why it matters: Falling outside this AI Act prohibition does not necessarily end the legal review of a manipulative interface. - Early Documented Prohibition Checks: Describe when to run a prohibition check and what to record if a proposed system appears close to, but outside, a ban.
Why it matters: An early, recorded decision gives the team a basis for stopping a banned project or defending a reasoned boundary judgment.
Key terms: Article 5, prohibited practices, emotion inference, biometric data, social scoring, guest trustworthiness score, biometric categorization, sensitive attributes, demographic fairness, algorithmic proxy
7.2.3 High Risk Classification
This section explains how an AI system’s intended use determines high-risk classification under the EU AI Act, including exceptions, borderline cases, and changes in agent capabilities.
- Purpose Based Risk Classification: Explain why the same model can receive different risk classifications in different deployments, and describe how an intended use in an Annex III domain can make a system high-risk.
Why it matters: A change in use can change your obligations even when the technology stays the same. - Self Assessment and Profiling Limits: Describe when a provider may justify treating an Annex III system as not high-risk, why genuine independent human review matters, and how profiling limits that option.
Why it matters: A claimed exception must hold up in actual use, especially when people may defer to AI output. - Regulated Product Safety Components: Describe the other high-risk classification path for an AI safety component in a regulated physical product and how it relates to the product’s existing conformity assessment.
Why it matters: Annex III use cases are not the only route to high-risk status. - Consequential Use Gray Zones: Apply the section’s three questions to distinguish a policy explanation from a potentially high-risk recommendation, and describe what to record when the answer is uncertain.
Why it matters: The line between explaining a policy and influencing an individual outcome can change a system’s classification. - Controls During Classification Uncertainty: Justify implementing controls for consequential decisions before a high-risk classification is resolved.
Why it matters: Classification uncertainty need not delay protections for consequential uses. - Agent Classification and Drift: Explain how agents are classified and why their classification must be revisited as their tools and actions change.
Why it matters: An agent’s actual reach can change its risk classification even when its model stays unchanged.
Key terms: intended purpose, Annex III, high-risk, credit scoring, Art. 6(3) self-assessment, profiling, decision support, automation bias, Annex I, safety element
7.2.4 General Purpose AI (GPAI) Models
This section separates GPAI model-provider duties from downstream system duties and explains what builders should check when choosing and using GPAI models.
- Model and system obligations: Distinguish a GPAI model provider’s compliance duties from the duties of an organization building a system on that model.
Why it matters: A compliant model provider does not make your application compliant automatically. - Downstream documentation evidence: Describe the provider information needed for a downstream compliance case and how to obtain and maintain it, whether the model is integrated directly or sits behind another vendor’s application.
Why it matters: Your compliance case depends on evidence you can obtain and keep current, not merely on a vendor’s assurance. - Code of Practice status: Explain what the GPAI Code of Practice covers and what signing or not signing means for provider evaluation.
Why it matters: A vendor’s Code status informs your evaluation, but neither signing nor declining to sign settles your own obligations. - GPAI provider boundary: Distinguish ordinary downstream customization from offering a modified model to others as a general-purpose model.
Why it matters: Customizing a model for an application and providing a new GPAI model are different regulatory positions. - Open-source exemption limits: State the limits of the partial open-source exemption and the licensing question raised by third-party fine-tuning data.
Why it matters: Calling a model open source does not, by itself, resolve monetization or training-data licensing concerns. - Systemic-risk model distinction: Explain the higher systemic-risk GPAI tier, its extra provider duties, and what using such a model means for a downstream system.
Why it matters: You can scrutinize a high-impact model provider’s additional safeguards without misclassifying your own system.
Key terms: GPAI model provider, risk assessment, oversight, monitoring, technical documentation, integration guidance, evidence pack, AI Office, GPAI Code of Practice, presumption of conformity
7.2.5 Transparency and AI Literacy Obligations
This section explains EU AI Act transparency and AI literacy duties that apply across risk tiers, then uses ClaimAssist to show how those duties differ from high-risk classification.
- AI Interaction Disclosure: Explain when a person must be told they are interacting with AI, who owns the disclosure, and how the narrow obvious-context exception works.
Why it matters: Even a limited-risk chatbot needs a disclosure that users actually encounter. - Generated Media Metadata: Describe the machine-readable marking required for AI-generated images, audio, and video, including how responsibility changes when an organization builds its own generation tool.
Why it matters: Using another company’s model does not necessarily make that company responsible for marking media your tool generates. - Visible Deepfake Labels: Explain when published deepfakes need a human-readable label and how that duty differs from embedded media marking.
Why it matters: File metadata alone does not tell a person viewing a realistic fake that it is synthetic. - Biometric Inference Disclosure: Distinguish biometric-based emotion recognition or categorization that requires advance disclosure from a keyword-based classifier, including when a vendor supplies the feature.
Why it matters: A built-in vendor feature can create a disclosure duty that is easy to overlook. - Role-Specific AI Literacy: Explain the AI literacy duty for organizations using AI and why training should reflect staff roles and the AI tools actually in use.
Why it matters: Training cannot prepare staff for AI risks if it ignores their work or the tools they use. - ClaimAssist Risk Threshold: Explain why the FAQ version of ClaimAssist has transparency duties without being high-risk, and what change would cross the high-risk threshold.
Why it matters: Changing a model does not change a use case’s classification, while changing what the system does can.
Key terms: Article 50, deployer, disclosure, machine-readable metadata, C2PA, marking, deepfakes, human-readable label, emotion recognition, biometric categorization
7.3 Roles and Duties
This section uses a creditworthiness system to show that EU AI Act roles follow an organization’s responsibilities, not its contract title, and that one organization can hold different roles for different systems.
- Responsibility-Based Provider Status: Explain why the financial services client is the provider of the creditworthiness system despite the contract calling it a deployer, and how that left it exposed when required documentation was missing.
Why it matters: A contract title will not protect an organization from duties attached to what it actually does with an AI system. - System-Specific Organizational Roles: Describe how an organization can be a provider for one AI system and a deployer of another, using the bank example.
Why it matters: Checking roles system by system avoids assigning one blanket role to an organization.
Key terms: provider, deployer, supply-chain duties, put the system into service
7.3.1 Provider, deployer, and other roles
This section explains how AI Act roles attach to organizations that build, configure, use, or supply AI systems, and what those roles mean in practice.
- Provider status and risk tiers: Explain when an organization is a provider, including when it puts an internal system into use, and how the provider’s obligations differ by risk tier.
Why it matters: Provider duties may attach even when you never sell a standalone AI product. - Application ownership versus configuration: Distinguish building a specific-purpose system from a model API or vendor components from configuring a vendor’s product within its intended purpose.
Why it matters: Using an external model or changing a product’s settings does not, by itself, settle your regulatory role. - Deployer oversight and system fit: Describe a deployer’s operational duties and why it must check that a system is used within the provider’s boundaries.
Why it matters: Being a deployer still requires active oversight after a system goes live. - Incident reporting and impact assessment: State when a deployer must report a serious incident, whom it notifies, and which deployers must assess fundamental-rights impacts before first use.
Why it matters: A deployer needs reporting routes ready before an incident, and some uses require assessment before launch. - Other AI supply chain roles: Distinguish the checks performed by importers and distributors from the contact-point function of an authorized representative.
Why it matters: These roles help close compliance gaps between a provider and the EU market.
Key terms: provider, placing on the market, put into service, risk tier, deployer, intended purpose, GPAI model provider, integration layer, instructions for use, human reviewer
7.3.2 When a deployer becomes a provider
This section explains three ways an organization using someone else’s AI system can become its provider.
- Branding a high-risk system: Explain why putting your name or trademark on an existing high-risk AI system and making it available to others makes you its provider.
Why it matters: Selling a third-party system under your own brand can change your regulatory role. - Changing the intended purpose: Explain how using a system for a new purpose that makes it high-risk can turn its deployer into its provider.
Why it matters: A vendor’s testing for one use does not cover an organization’s newly introduced high-risk use. - Substantially modifying the system: Describe how a substantial modification can make the organization the provider of a modified AI system, using the bank’s automation example.
Why it matters: Automation can change a system beyond the use its supplier validated, shifting the provider role.
Key terms: provider, deployer, high-risk AI system, new intended purpose, high-risk, substantial modification, compliance, intended purpose
7.3.3 What high-risk providers must do
This section explains the system-wide, documentation, pre-launch, and continuing obligations of providers of high-risk AI systems.
- Scope of the regulated system: Explain why high-risk obligations apply to the system as provided for its intended purpose, not just to its predictive model. Describe the components included in ClaimAssist V5.
Why it matters: The boundary you draw around the system determines what must meet the provider’s obligations. - Internal assessment and current evidence: Explain why providers of credit models and most Annex III systems must prepare official technical documentation before launch and keep it current.
Why it matters: Without a certifier’s sign-off, the organization must be ready to show its own compliance evidence. - Reviewable documents and deployer instructions: Distinguish the technical documentation package from the instructions for use, and describe what each must enable its reader to do.
Why it matters: Strong technical work is not enough if authorities cannot review its evidence or deployers cannot use the system safely. - Pre-launch steps and quality processes: State the pre-launch conformity steps and describe how a quality management system supports continuing compliance.
Why it matters: A launch checklist and repeatable quality processes make compliance an operational responsibility rather than a one-time claim. - Monitoring and response duties: Describe how a provider must monitor a high-risk system after launch and respond when monitoring identifies a problem or a serious incident.
Why it matters: Provider obligations continue after release, when real-world evidence may require urgent action. - Regulator access and record retention: Describe what authorities may request and distinguish the retention rules for compliance documents and automatically generated logs.
Why it matters: A provider needs to be able to answer an authority’s request even after the system has been decommissioned.
Key terms: AI system, declared intended purpose, ClaimAssist V5, internal self-assessment, technical documentation, market surveillance authority, instructions for use, intended purpose, human oversight, conformity assessment
7.3.4 What high-risk deployers must do
This section explains the duties a high-risk AI deployer must meet before signing, before first use, and while the system operates—duties that the provider’s compliance does not cover.
- Separate deployer duties and contracts: Explain why provider compliance does not cover the deployer’s duties, and describe the documentation and contractual access a deployer should secure before signing.
Why it matters: Signing without certification, instructions, and operational access can leave you unable to meet your own duties. - Validated inputs and populations: Explain why a provider’s accuracy and fairness claims may not hold for a deployer’s chosen inputs or population, and state the options when deployment falls outside the validated context.
Why it matters: A system can be deployed exactly as supplied yet perform poorly—or unfairly—in your setting. - Effective human oversight: Describe what designated overseers of a high-risk system must be able and authorized to do, and distinguish that from routinely clicking through outputs.
Why it matters: Oversight cannot protect people if reviewers lack the time, judgment, or authority to challenge the system. - Notices and individual responses: Describe whom a high-risk deployer must inform and the two kinds of individual response it should prepare before launch.
Why it matters: Affected people need notice and usable ways to question significant decisions involving AI. - Fundamental rights impact assessment: State which deployers must complete a FRIA before first use and explain how the section suggests coordinating it with existing assessments.
Why it matters: Coordinating overlapping assessments can meet the pre-use requirement without duplicating the same work. - Operational monitoring and response: Describe what a deployer should do when monitoring reveals reasonable grounds for risk, alongside its log-retention and authority-cooperation duties.
Why it matters: Monitoring only helps if the deployer can act on deterioration and preserve the records authorities may need.
Key terms: deployer, provider, CE marking, declaration of conformity, instructions for use, accuracy and fairness claims, input design, validation, population, operating conditions
7.3.5 What your model provider owes you
This section explains what documentation and cooperation you can expect from AI suppliers, how supply-chain arrangements limit those expectations, and what to secure in contracts when deploying their products.
- Supply Chain Documentation Rights: Distinguish the documentation owed to a system provider integrating a GPAI model from the instructions owed to a deployer of a finished high-risk product. Explain how to check delivery against the GPAI Code of Practice and whom to contact when a provider is outside the EU.
Why it matters: Your place in the supply chain determines what evidence you can demand and whom you can hold accountable. - Open Source Documentation Limits: Explain what an open-source GPAI provider must still disclose, what documentation it need not provide on request, and why a downstream high-risk user may have to fill the remaining gap.
Why it matters: An open model card can help, but it does not automatically supply the evidence a high-risk downstream use needs. - Layered Vendor Contract Boundaries: Explain why a customer of a SaaS product must seek documentation and data protections through its direct vendor, rather than assuming it can enforce terms against the underlying model provider.
Why it matters: A familiar model name behind a SaaS product does not give you its documentation or its data-handling terms. - Missing Provider Evidence Responses: Describe the options when a provider will not supply needed evidence and what must be recorded if deployment proceeds despite the gap.
Why it matters: A documentation gap calls for an explicit response, not an undocumented decision to proceed. - Model Evidence and Version Terms: Describe the technical information and model-version commitments a supplier contract should secure, including why the update-notice window must fit re-validation.
Why it matters: Evidence about a model is less useful if an unannounced update changes the model before you can re-validate it. - Logs and Compliance Allocation: Explain how a supplier contract should secure usable logs, allocate compliance work, and require notice of serious incident reports.
Why it matters: Without explicit terms, you may be unable to retain your logs or learn promptly that your supplier has reported a serious problem.
Key terms: system provider, deployer, GPAI Code of Practice, authorized representative, open-source GPAI models, training data summary, copyright compliance policy, systemic risk, model card, layered providers
7.3.6 Finding yourself on the map
This section helps teams identify their AI Act role in commonly misclassified deployments and reassess role and risk as a system changes.
- Fine-Tuning and Provider Status: Distinguish fine-tuning a model from the acts that can make an organization a provider, including releasing a heavily fine-tuned GPAI model or substantially modifying a deployed high-risk system.
Why it matters: The amount of model customization alone is an unreliable way to assign regulatory responsibility. - New Purpose Through RAG: Explain why adding retrieval to a model API can still make the organization the provider of a new AI system when it gives the model a specific high-stakes purpose.
Why it matters: Using someone else's model API does not necessarily leave provider obligations with the model vendor. - Agent Platform Role Ownership: Explain how a team using an agent-building platform can be the provider of the resulting agent system despite relying on a turnkey vendor platform.
Why it matters: A turnkey interface can obscure who is responsible for the agent system built on it. - Combined Classification Review: Describe how and when a team should assess both a system's risk classification and its provider or deployer role, and where it should record the result.
Why it matters: A one-time classification may no longer match the system a team actually operates. - Uncertain Roles and AI Literacy: Describe what a team should do when its provider or deployer role is unclear, and state which duty applies regardless of the classification path.
Why it matters: Ambiguity about a role does not justify postponing basic governance or staff competence.
Key terms: fine-tuning, provider, GPAI model, substantial modification, RAG, model API, new use, agent-building platforms, component supplier, risk classification
7.4 What you already have and what’s still missing
An ISO 42001 certificate shows an AI management system is in place, but does not establish compliance with the AI Act’s provider- and deployer-specific requirements.
- Management Certification Versus AI Act Compliance: Explain why an ISO 42001 certificate does not establish AI Act compliance, and distinguish the additional requirements the section identifies for providers and deployers.
Why it matters: A management-system certificate cannot replace evidence that the organization has met its particular AI Act duties.
Key terms: ISO 42001, AI management system, AI Act, provider, deployer
7.4.1 Are your existing frameworks enough?
Existing AI governance frameworks provide a head start, but high-risk providers and deployers must still meet more specific EU AI Act requirements.
- Framework Head Start and Limits: Explain why ISO 42001 or the NIST AI RMF helps an organization prepare for AI Act compliance without being sufficient on its own.
Why it matters: An existing framework is a useful foundation, not proof that every AI Act obligation has been met. - Remaining High-Risk Provider Duties: Describe the provider obligations ISO 42001 does not supply, distinguishing formal compliance steps from the AI Act’s more specific quality-management and deployer-documentation requirements.
Why it matters: A provider must translate its existing management system into the Act’s formal steps, required processes, and deployer-facing information. - Act-Specific Evidence Content: Explain why ISO-style documented evidence must be supplemented with content tailored to the AI Act and the system’s intended purpose or use case.
Why it matters: Having an evidence document is not enough if it omits the analyses the Act calls for. - Uncovered High-Risk Deployer Obligations: State the deployer-specific AI Act obligations that ISO 42001 does not cover and explain that its coverage of deployer activity is weaker than its provider coverage.
Why it matters: Deployers need a separate gap check rather than assuming a provider-oriented framework covers their responsibilities.
Key terms: ISO 42001, NIST AI RMF, EU AI Act, high-risk providers, conformity assessment, CE marking, registration, quality management system, instructions for use, documented evidence
7.4.2 The evidence pack for providers
This section explains how providers assemble and maintain an EU AI Act evidence pack, complete its Act-specific artifacts, and divide responsibility for the technical and legal work.
- Reviewable provider evidence pack: Explain why a provider needs a coherent evidence pack rather than a vendor contract, certification, or scattered tickets, and why it must be assembled during development and retained after launch.
Why it matters: A reviewer needs a usable record of decisions and evidence even if the original team or system is gone. - High-risk conformity assessment routes: Describe the usual conformity assessment for a high-risk system and the two exceptions described here.
Why it matters: The assessment route determines who reviews the provider’s evidence before conformity is declared. - Act-specific provider artifacts: Describe what CE marking, EU database registration, and the declaration of conformity do in the provider’s compliance work.
Why it matters: These formal artifacts communicate compliance, but they do not replace the evidence that justifies it. - Technical and legal evidence ownership: Distinguish the technical team’s evidence responsibilities from the lawyer’s regulatory responsibilities, and describe how a single owned repository supports their handoff.
Why it matters: Clear ownership makes the compliance file both technically substantiated and legally usable. - Documented joint risk classification: Explain why high-risk classification requires both technical and legal input and what the lasting classification record should contain.
Why it matters: A recorded rationale makes a consequential classification decision defensible and easier to revisit when features change.
Key terms: evidence pack, retention obligation, conformity assessment, notified body, harmonized standards, regulated product, CE marking, EU database, Declaration of Conformity, technical evidence
7.4.3 The evidence pack for deployers
This section explains what deployers must be able to document and how to close evidence gaps in systems already in use.
- Deployer Evidence Responsibility: Distinguish a deployer’s evidence trail from a builder’s: a deployer must show that it selected the right system, configured it for its intended use, and operated it as the provider designed.
Why it matters: It keeps the deployer’s evidence effort focused on its own responsibilities. - Retention and Logging Evidence: Explain why a provider’s short API log retention may leave a deployer unable to meet the six-month obligation and why logging architecture must be decided before go-live.
Why it matters: A retention gap exists from launch if the deployer relies on shorter-lived provider logs. - Proof of Actual Operation: Distinguish written oversight and monitoring plans from evidence that those plans have been carried out.
Why it matters: A regulator needs evidence that controls operated, not just a statement that they were planned. - Shadow AI Inventory Gaps: Explain why undiscovered AI use prevents a deployer from producing evidence, including AI added through SaaS tools or adopted within business-unit projects.
Why it matters: Finding AI already in use is a prerequisite to assembling its evidence. - Existing System Evidence Recovery: Describe how to address an AI system already in production without adequate documentation: record its current state and gaps honestly, gather evidence available now, and prioritize the highest-risk systems.
Why it matters: It offers a truthful way to improve documentation for systems that launched without it.
Key terms: deployer, evidence trail, intended use, conformity assessment, CE mark, logs, provider’s API, independent logging, six-month obligation, go-live
7.5 Living Compliance
This section explains why compliance must continue after launch: system changes can invalidate earlier assessments and evidence, while providers and deployers have ongoing duties.
- Compliance After System Changes: Explain how a post-launch model update can make an earlier risk assessment and evidence pack outdated, even after a system has completed its initial compliance steps.
Why it matters: An evidence pack is only useful if it still describes the system that is running. - Ongoing Provider and Deployer Duties: Distinguish the post-deployment duties stated for providers and deployers, and describe when a significant system change may call for reassessment.
Why it matters: Post-launch responsibilities apply to both roles, rather than ending when the system is released.
Key terms: risk assessment, evidence pack, accuracy metrics, providers, deployers, serious incidents, reassess
7.5.1 Post-market monitoring
This section explains how providers and deployers monitor high-risk AI systems after launch, when to reassess them, and how to handle monitoring logs that may contain personal data.
- Built-in monitoring capabilities: Explain which capabilities a provider must build into a high-risk system before launch and why documentation alone is insufficient.
Why it matters: Monitoring and oversight require capabilities in the deployed product, not just promises on paper. - Provider monitoring plan: Describe the provider's post-market monitoring plan and distinguish it from the QMS.
Why it matters: A system-specific plan makes production monitoring actionable rather than merely an organizational policy. - Deployer monitoring responsibilities: Distinguish the deployer's monitoring work from the provider's monitoring-plan responsibility.
Why it matters: The provider's testing cannot replace observation of the system in each deployer's actual setting. - Drift and reassessment triggers: Explain what monitoring is meant to catch, when reassessment is warranted, and how documented expected changes affect re-filing.
Why it matters: A functioning system can become less reliable without an obvious failure or a provider update. - Personal data in logs: Explain how to reconcile monitoring logs with privacy requirements, including deletion requests and free-text prompt logs.
Why it matters: Useful monitoring records can also expose people, so their collection, retention, and access need a defensible purpose.
Key terms: automatic event logging, human oversight, override controls, instructions for use, post-market monitoring system, monitoring plan, technical documentation, QMS, deployers, oversight measures
7.5.2 Incident reporting
This section explains how to recognize, report, and respond to serious AI incidents under the EU AI Act, including how that process differs from GDPR breach reporting.
- Serious incident threshold: Explain what makes an AI failure a potentially serious incident and apply the section’s harm, pattern, and persistence questions to ClaimAssist.
Why it matters: The distinction determines which failures need serious-incident reporting rather than ordinary quality handling. - Reporting tiers and clock: Describe the three AI Act reporting deadlines and when their clocks begin.
Why it matters: Waiting for a definitive root-cause finding can consume time the reporting rules do not grant. - Deployer and provider duties: Distinguish the deployer’s immediate notification duty from the provider’s formal reporting and corrective-action duties.
Why it matters: Neither party can assume the other’s reporting role removes its own duty to act. - AI Act and GDPR reporting: Distinguish AI Act serious-incident reporting from GDPR breach reporting and explain what happens when one event triggers both.
Why it matters: An existing privacy-breach workflow is a useful starting point, but it cannot replace AI incident reporting. - Prepared incident reporting workflow: Describe how to prepare a reporting workflow before an incident and handle an investigation that is unfinished at the deadline.
Why it matters: Advance preparation leaves the reporting window for investigation rather than figuring out who to call.
Key terms: serious incident, fundamental rights, systemic failure, reporting clock, causal link, reasonable likelihood, market surveillance authority, deployer, provider, corrective action
7.5.3 Regulatory sandboxes and SME provisions
This section explains how regulatory sandboxes and smaller-organization provisions can ease the path to compliance without reducing AI Act obligations.
- Sandbox benefits and limits: Explain how a regulatory sandbox lets providers and deployers test AI systems under supervision before launch, and why its results can strengthen a later evidence pack without replacing conformity assessment. Describe why an organization should not wait for a sandbox if none is available.
Why it matters: A sandbox can help you find gaps early, but it is not a compliance shortcut or a reason to delay. - Smaller organization relief: Distinguish the AI Act’s penalty calculation for SMEs and startups from that for larger companies, and explain the proposed small mid-cap relief. State why smaller organizations still owe the same substantive high-risk compliance work.
Why it matters: Lower penalties or cheaper ways to prepare do not exempt a small organization from high-risk AI requirements.
Key terms: regulatory sandboxes, conformity assessment, evidence pack, market surveillance authorities, SMEs, startups, small mid-caps, penalty tiers, Digital Omnibus
7.6 ClaimAssist across the risk thresholds
ClaimAssist illustrates how provider status, risk classification, and governance evidence together determine a team’s EU AI Act exposure.
- Provider Role and Risk Classification: Distinguish ElectraMotors’ provider role from ClaimAssist’s risk classification, and explain what changed at V5.
Why it matters: A change in obligations does not necessarily mean your organization has taken on a new role. - Traceable Governance Evidence: Explain how earlier governance work affects the effort of assembling an AI Act evidence pack.
Why it matters: Producing traceable artifacts during governance work makes compliance evidence easier to assemble when obligations arise.
Key terms: provider, high-risk, classification, obligations, technical evidence, traceable artifacts, evidence pack
7.7 Summary
The section distills EU AI Act compliance into system classification, organizational role, awareness of changes, and traceable product-level evidence.
Where this is used later
- The road ahead (chapter 8) · Regulatory roles and duties
Concepts explained in this chapter
- deployerA regulatory role for an entity that uses an AI system under its authority, except where the system is used in the course of a personal, non-professional activity.
- providerUnder the EU AI Act, a provider is a person or organization that develops or has an AI system developed and places it on the market or puts it into service under its own name or trademark.
- AI ActEuropean Union legislation establishing a risk-based regulatory framework for AI systems and obligations for their providers and deployers.
- serious incidentA serious incident is an event that directly or indirectly causes death, serious health damage, specified critical-infrastructure disruption, a breach of fundamental-rights obligations, or serious damage to property or the environment.
- documentationTechnical documentation is recorded information describing a system’s design, capabilities, limitations, and compliance-relevant characteristics.
- self-assessmentAn evaluation in which a provider assesses its own system against applicable requirements rather than relying on an external reviewer.
- intended purposeIntended use is the purpose for which a system is meant to be used, as specified by its provider or deployer.
- biometric categorizationBiometric categorization is the classification of people into categories using physical or behavioral biometric data.
- compliance evidenceCompliance evidence is documented information supporting an assessment of whether a system meets applicable requirements.
- risk tierA defined classification grouping an AI use case by the severity and acceptability of its risks.
- monitoring planA documented plan specifying how post-market information about an AI system will be collected and evaluated.
- post-market monitoringThe ongoing collection and review of information about a system after it has been placed on the market or put into service.
- substantial modificationA substantial modification is a change to an AI system that affects its compliance or changes its intended purpose in a way not foreseen by the original provider.
- ai literacyThe knowledge and skills that enable providers, deployers, and other people involved with AI systems to make informed use of them and understand their opportunities and risks.
- artificial intelligence systemAn AI system is a machine-based system that operates with varying levels of autonomy and may adapt after deployment to generate outputs such as predictions, recommendations, or decisions.
- AI Code of PracticeThe GPAI Code of Practice is a voluntary code that provides guidance for providers of general-purpose AI models on meeting relevant obligations and commitments.
- open-source GPAI modelAn open-source GPAI model is a general-purpose AI model made available under open-source terms whose provider may qualify for specified regulatory exemptions.
- version pinningVersion pinning is the practice of fixing software or model use to a specified version rather than automatically adopting updates.
- incident reporting obligationAn incident reporting obligation is a legal duty to notify designated authorities about a qualifying incident within a specified period.
- profilingAutomated processing of personal data used to evaluate or predict aspects of an individual's behavior, preferences, or circumstances.
- general-purpose AI providerA general-purpose AI provider is an entity that develops or places a general-purpose AI model on the market under its name or trademark.
- high-risk providerA provider responsible for placing a high-risk AI system on the market or putting it into service and meeting the associated regulatory obligations.
- quality management systemAn organized set of documented processes and responsibilities for maintaining the quality of an organization's products or services.
- risk classificationClassification is the assignment of an AI system or use case to a regulatory category based on its purpose and applicable rules.
- manipulation and exploitation of vulnerabilitiesManipulation and exploitation of vulnerabilities are AI practices that influence people through manipulative tactics or by taking advantage of vulnerabilities such as age, disability, or financial circumstances.
- prohibited practiceA prohibited practice is an AI use that legislation bans outright rather than permitting subject to ordinary controls.
- social scoringSocial scoring is the creation of a score from a person’s behavior and use of that score to disadvantage or exclude them.
- systemic-risk modelA systemic-risk model is a general-purpose AI model classified at a heightened regulatory level because its scale or capabilities could create widespread effects.
- logLogging is the recording of system events and associated information for monitoring, audit, or analysis.
- shadow AIShadow AI is the unsanctioned use of artificial-intelligence tools or models outside an organization's official governance processes.
- instructions for useProvider-supplied directions describing a system's intended purpose, limitations, requirements, and conditions for use.
- authorized representativeUnder the AI Act, an authorized representative is an entity established in the European Union that receives and accepts a written mandate from a provider outside the Union to perform specified obligations on its behalf.
- placing on the marketPlacing on the market is making an AI system available in the European Union for the first time.
- putting into servicePutting an AI system into service is making it available for use for the first time for its intended purpose.
- AI agentAn AI agent is a software system that can plan and perform tasks, including using tools and taking actions on a user's behalf.
- accuracyAccuracy is the data-protection principle that personal data be correct and, where necessary, kept up to date.
- purpose limitationPurpose limitation is the data-protection principle that personal data may be collected for specified, explicit, and legitimate purposes and not processed incompatibly with them.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
