Model risk arises when an application depends on a model the deploying organization has not fully vetted or does not control. Problems in the model or its customization can affect everything built on top of it. section 4.2 section 4.2.1
Assess model risk by checking whether the model:
- Was trained on malicious instructions or uses open-source weights not evaluated for safety and integrity. section 4.2
- Is vendor-hosted, so behavior changes after updates cannot be inspected. section 4.2
- Was fine-tuned using data that may be poisoned or biased, weaken safety guardrails, or cause sensitive information to be memorized and leaked. section 4.2 section 4.2.1
ClaimAssist
An unvetted model adds model risk to ClaimAssist’s FAQ bot. Fine-tuning its assistant adds further concerns: weakened refusal behavior and possible leakage of memorized claims or internal notes. section 4.2.1
Table 4.1 How each additional ClaimAssist feature increases the security risk
|
Version
|
Environment
|
Input
|
Model
|
Data
|
Ability to Act
|
Autonomy
|
|
V1: FAQ Bot
|
Yes
|
Yes
|
No
|
No
|
No
|
No
|
|
V2: FAQ Bot
running on an unvetted model
|
Yes
|
Yes
|
Yes
|
No
|
No
|
No
|
|
V3:
Personalized Assistant
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
No
|
|
V4:
Fine-tuned Assistant
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
No
|
|
V5:
Action-Enabled Assistant
|
Yes
|
Yes
|
Yes
|
Yes
|
Yes
|
No
|
|
V6: Agentic
Assistant
|
Yes
|
Yes
|
Yes
|
Yes
|
Yes
|
Yes
|
Compare how model risk appears alongside other factors as ClaimAssist gains features. section 4.2.1
