liveBook · Manning

Regression test

As explained in LLM Customization and Fine-Tuning

A test that checks whether a change has caused previously acceptable behavior or performance to deteriorate.

A regression test checks whether a change has weakened previously acceptable behavior. For fine-tuned models, compare responses to a fixed set of safety and refusal prompts before and after the update. section 5.11

Check refusal behavior

Run the same safety and refusal prompts against both models. If the fine-tuned model becomes more permissive in harmful ways, consider adjusting the learning rate, adding refusal examples, or mixing in base-model data. section 5.11

Table 5.9 Responsible AI risks and controls in fine-tuning
Area
Key Risk
What to Watch For
Typical Mitigation
Privacy and Compliance
Data leakage, regulatory violations
PII in training data, unclear data ownership
Anonymization, consent checks, and data residency controls
Fairness and Bias
Discriminatory or uneven outputs
Skewed examples, demographic imbalance
Dataset rebalancing, bias tests, targeted evaluations
Safety
Over-compliance, harmful responses
Loss of refusal behavior after fine-tuning
Safety prompt sets, refusal examples, and comparative testing
Evaluation
False confidence in model quality
Overreliance on a single metric
Mixed metrics, human review, segmented analysis
Transparency
Misuse or misinterpretation
Unclear limitations or intended use
Model cards, documentation, usage boundaries
Monitoring
Degradation over time
New failure modes, user complaints
Logging, feedback loops, iterative LoRA updates

Compare the safety controls with other fine-tuning risks and evaluation practices. section 5.11

How LLM Customization and Fine-Tuning covers it

How deep it goes
Compares fixed prompts before and after fine-tuning and identifies mitigation options for harmful permissiveness.
Examples
Uses a fixed set of safety and refusal prompts to compare model behavior across fine-tuning.
What you can do after
Evaluate whether fine-tuning has made a model more permissive in harmful ways.
How it connects
Links fine-tuning evaluation to safety prompts, refusal behavior, and model safeguards.

Also explained in

Compare how 3 books explain regression test →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.