liveBook · Manning

Activation difference

As explained in Rearchitecting LLMs

A signed activation difference is the stereotyped-prompt mean activation minus the majority-prompt mean activation for a neuron.

A signed activation difference is the mean activation for a stereotyped prompt minus the mean activation for a majority prompt, calculated separately for each neuron. Positive and negative values preserve the direction of the difference. section 11.2.1

To compare a pair of prompts:

  • Capture activations at the chosen model point for each prompt. section 11.2.1
  • Average across batch and token dimensions to get one value per neuron, then subtract the majority-prompt vector from the stereotyped-prompt vector. section 11.2.1
  • Keep the mean activation signs for both prompts as well; the difference alone can make distinct sign patterns look similar. section 11.2.1
Listing 11.5 Compute MagDiff
def compute_signed_bias(
    model,
    tokenizer,
    prompt_majority: str,
    prompt_stereotyped: str,
    target_layers: Optional[List[str]] = None,
):
    if target_layers is None:
        target_layers = [CAPTURE_POINT] #A
 
    acts_stereotyped = get_prompt_activations(
                            model, 
                            tokenizer,
                            prompt_stereotyped,
                            target_layers=target_layers
                         ) #B
    acts_majority= get_prompt_activations(
                            model, 
                            tokenizer,
                            prompt_majority,
                            target_layers=target_layers
                         ) #B
 
    result: Dict[str, torch.Tensor] = {}
    for layer_key in acts_majority:
        maj_tensor = acts_majority[layer_key].float()
        ster_tensor = acts_stereotyped[layer_key].float()
 
        reduce_dims_maj = tuple(range(maj_tensor.ndim - 1)) #1
        reduce_dims_ster = tuple(range(ster_tensor.ndim - 1)) #1
 
        maj_pooled = maj_tensor.mean(dim=reduce_dims_maj)
        ster_pooled = ster_tensor.mean(dim=reduce_dims_ster)
 
        magnitude_diff = ster_pooled - maj_pooled #2
 
        maj_sign = torch.sign(maj_pooled)
        ster_sign = torch.sign(ster_pooled) #3
 
        result[layer_key] = (
            magnitude_diff.cpu(), 
            maj_sign.cpu(), 
            ster_sign.cpu()) #4
 
    return result

The PyTorch function captures both prompt activations and computes per-neuron differences. section 11.2.1

Using differences to guide an intervention

For one prompt pair, neurons were ranked by the magnitude of their differences, and five were scaled. Their differences fell to about one tenth of the original values. On a different scenario, shrinking the difference did not specify which output should prevail: both prompts produced the same profession after intervention. section 11.2 section 11.2.5 section 11.2.6

How Rearchitecting LLMs covers it

How deep it goes
Calculates per-neuron means and signed differences, while noting that the metric alone does not determine which output an intervention favors.
Examples
Compares paired majority and stereotyped prompts, then checks differences after scaling and tests another scenario.
What it assumes
Assumes paired majority and stereotyped prompts and captured activations; it averages across tokens rather than aligning individual tokens.
What you can do after
Apply via PyTorch to calculate per-neuron differences and compare them before and after scaling.
How it connects
Connects neuron activations and MLP capture points to neuron selection and model intervention.

This explainer was generated by Manning's knowledge graph from the text of Rearchitecting LLMs. The section links point to the parts of the book it draws on.