liveBook · Manning

Model risk

As explained in AI Governance

The risk of relying on a model whose training and development are outside the deploying organization’s control.

Model risk arises when an application depends on a model the deploying organization has not fully vetted or does not control. Problems in the model or its customization can affect everything built on top of it. section 4.2 section 4.2.1

Assess model risk by checking whether the model:

  • Was trained on malicious instructions or uses open-source weights not evaluated for safety and integrity. section 4.2
  • Is vendor-hosted, so behavior changes after updates cannot be inspected. section 4.2
  • Was fine-tuned using data that may be poisoned or biased, weaken safety guardrails, or cause sensitive information to be memorized and leaked. section 4.2 section 4.2.1

ClaimAssist

An unvetted model adds model risk to ClaimAssist’s FAQ bot. Fine-tuning its assistant adds further concerns: weakened refusal behavior and possible leakage of memorized claims or internal notes. section 4.2.1

Table 4.1 How each additional ClaimAssist feature increases the security risk
Version
Environment
Input
Model
Data
Ability to Act
Autonomy
V1: FAQ Bot
Yes
Yes
No
No
No
No
V2: FAQ Bot running on an unvetted model
Yes
Yes
Yes
No
No
No
V3: Personalized Assistant
Yes
Yes
Yes
Yes
No
No
V4: Fine-tuned Assistant
Yes
Yes
Yes
Yes
No
No
V5: Action-Enabled Assistant
Yes
Yes
Yes
Yes
Yes
No
V6: Agentic Assistant
Yes
Yes
Yes
Yes
Yes
Yes

Compare how model risk appears alongside other factors as ClaimAssist gains features. section 4.2.1

How AI Governance covers it

How deep it goes
Identifies model and fine-tuning failure modes and consequences, but does not specify a model-vetting procedure.
Examples
Contrasts an FAQ bot using an unvetted model with a fine-tuned assistant that may weaken refusals or leak memorized data.
What you can do after
Evaluate model exposure by identifying control gaps, update visibility limits, and fine-tuning risks that warrant additional controls.
How it connects
Connects model risk to model supply-chain issues, fine-tuning, safety alignment, and data leakage.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of AI Governance. The section links point to the parts of the book it draws on.