liveBook · Manning

Refusal rate

As explained in LLM Customization and Fine-Tuning

Refusal rate is the proportion of tested requests that a model refuses, with the request set determining whether it measures appropriate or harmful-request refusals.

A model can improve on a task while becoming less willing to refuse harmful requests. Refusal rate is the proportion of tested requests the model refuses: refusals divided by total requests, expressed as a percentage. Its meaning depends on the request set. On harmful prompts, a high safety refusal rate indicates more refusals; on ordinary IT-support questions, refusals may instead be unwanted. section 1.8 section 6.3.2

For a release decision, compare the adapted model with its base model on the same fixed safety suite. One gate blocks release when harmful-request refusal rate drops by more than 10 percentage points. A rate below 80% is a red flag; well-aligned models are expected to maintain at least 90%. These thresholds do not replace the baseline comparison. section 1.8 section 5.1.4

Use the metric alongside behavioral and task checks:

  • Keep harmful-request tests separate from ordinary task questions. A 0% refusal rate on a held-out IT-support set means no requests were refused there; it does not establish how the model handles harmful prompts. section 6.3.2
  • Inspect individual regressions as well as the aggregate. A distilled student lost refusals on two harmful requests that the base model refused, making deployment unsafe despite other passing checks. section 7.8.2
  • Measure before and after preference optimization, even when training uses refusal examples: safety-related preference training can reduce refusal rates. section 8.8.2

An IT-support adapter fails the safety gate

In a 450-example LoRA training run, the base model refused all ten harmful prompts, while the adapter refused six: a drop from 100% to 60%, or 40 percentage points. Token-F1 rose only from 0.149 to 0.154. The safety regression blocks release under the 10-point gate, even though the training data contained no harmful examples. One mitigation is to add explicit refusal examples to training and rerun the gate. section 1.8 section 5.1.4

## base
- Overall exact match: 0.0%
- Overall token-F1: 0.149
- Safety refusal rate: 100.0%
## adapter
- Overall exact match: 0.0%
- Overall token-F1: 0.154
- Safety refusal rate: 60.0%
## adapter (Improvement vs Base)
- Token-F1 delta: +0.005
- Safety refusal rate delta: -40.0%
- Per-category token-F1 (base to adapter):
  - networking: 0.169 to 0.170 (+0.001)
  - security: 0.199 to 0.185 (-0.014)
  - windows: 0.133 to 0.141 (+0.008)
  - linux: 0.129 to 0.144 (+0.014)
  - hardware: 0.152 to 0.188 (+0.037)
  - software: 0.122 to 0.111 (-0.011)

Compare the adapter’s safety drop with its task-metric changes and category-level results. section 5.1.4

How LLM Customization and Fine-Tuning covers it

How deep it goes
Defines the percentage and provides comparison thresholds, regression checks, and evaluation code; treats the safety suite as an operational minimum.
Examples
Compares a base model’s 100% refusal rate with a LoRA adapter’s 60% on ten harmful prompts.
What you can do after
Diagnose safety regressions by comparing base and adapted models on a fixed harmful-request suite and applying a release gate.
How it connects
Connects safety regression to LoRA, supervised fine-tuning, knowledge distillation, and preference optimization.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Customization and Fine-Tuning. The section links point to the parts of the book it draws on.