Qualitative assessment examines model outputs directly instead of relying only on numerical scores. Human reviewers or LLM judges can assess qualities such as helpfulness, tone, and risk sensitivity—criteria that a single metric cannot capture. Reviewing results by relevant segments can also reveal weaknesses hidden by aggregate scores. section 5.11
Comparing VPN instructions
For a prompt about connecting a Mac to a company VPN, the base model gives generic advice, while the teacher model provides specific numbered steps. The distilled student keeps the structure but may omit details. Comparing these outputs reveals differences in actionability and specificity that Token-F1 alone misses. That gap may matter more for exact procedures than for general troubleshooting. section 7.7.1
Table 5.9 Responsible AI risks and controls in fine-tuning
|
Area
|
Key Risk
|
What to Watch For
|
Typical Mitigation
|
|
Privacy and Compliance
|
Data leakage, regulatory
violations
|
PII in training data,
unclear data ownership
|
Anonymization, consent
checks, and data residency controls
|
|
Fairness and Bias
|
Discriminatory or uneven
outputs
|
Skewed examples,
demographic imbalance
|
Dataset rebalancing, bias
tests, targeted evaluations
|
|
Safety
|
Over-compliance, harmful
responses
|
Loss of refusal behavior
after fine-tuning
|
Safety prompt sets,
refusal examples, and comparative testing
|
|
Evaluation
|
False confidence in model
quality
|
Overreliance on a single
metric
|
Mixed metrics, human
review, segmented analysis
|
|
Transparency
|
Misuse or
misinterpretation
|
Unclear limitations or
intended use
|
Model cards,
documentation, usage boundaries
|
|
Monitoring
|
Degradation over time
|
New failure modes, user
complaints
|
Logging, feedback loops,
iterative LoRA updates
|
Use the risk-and-control table to place human review alongside mixed metrics and segmented analysis when evaluating fine-tuned models. section 5.11

