A signed activation difference is the mean activation for a stereotyped prompt minus the mean activation for a majority prompt, calculated separately for each neuron. Positive and negative values preserve the direction of the difference. section 11.2.1
To compare a pair of prompts:
- Capture activations at the chosen model point for each prompt. section 11.2.1
- Average across batch and token dimensions to get one value per neuron, then subtract the majority-prompt vector from the stereotyped-prompt vector. section 11.2.1
- Keep the mean activation signs for both prompts as well; the difference alone can make distinct sign patterns look similar. section 11.2.1
Listing 11.5 Compute MagDiff
def compute_signed_bias(
model,
tokenizer,
prompt_majority: str,
prompt_stereotyped: str,
target_layers: Optional[List[str]] = None,
):
if target_layers is None:
target_layers = [CAPTURE_POINT] #A
acts_stereotyped = get_prompt_activations(
model,
tokenizer,
prompt_stereotyped,
target_layers=target_layers
) #B
acts_majority= get_prompt_activations(
model,
tokenizer,
prompt_majority,
target_layers=target_layers
) #B
result: Dict[str, torch.Tensor] = {}
for layer_key in acts_majority:
maj_tensor = acts_majority[layer_key].float()
ster_tensor = acts_stereotyped[layer_key].float()
reduce_dims_maj = tuple(range(maj_tensor.ndim - 1)) #1
reduce_dims_ster = tuple(range(ster_tensor.ndim - 1)) #1
maj_pooled = maj_tensor.mean(dim=reduce_dims_maj)
ster_pooled = ster_tensor.mean(dim=reduce_dims_ster)
magnitude_diff = ster_pooled - maj_pooled #2
maj_sign = torch.sign(maj_pooled)
ster_sign = torch.sign(ster_pooled) #3
result[layer_key] = (
magnitude_diff.cpu(),
maj_sign.cpu(),
ster_sign.cpu()) #4
return result
The PyTorch function captures both prompt activations and computes per-neuron differences. section 11.2.1
Using differences to guide an intervention
For one prompt pair, neurons were ranked by the magnitude of their differences, and five were scaled. Their differences fell to about one tenth of the original values. On a different scenario, shrinking the difference did not specify which output should prevail: both prompts produced the same profession after intervention. section 11.2 section 11.2.5 section 11.2.6
