Chapter 11: Precision pruning for bias
The introduction defines demographic bias, explains the basic idea of comparing prompts and adjusting a few neurons, and clarifies the limits of that technique.
After this chapter you will be able to
- Demographic bias in model responses. Explain that this chapter calls it bias when a model treats an otherwise identical case differently because a demographic attribute such as race or gender changes, and that internet-derived training data can contribute to unwanted responses.
- Contrastive comparison and targeted adjustment. Describe the chapter's basic approach: compare two prompts that differ only in a demographic variable to observe differing neuron activations, then adjust a few neurons' weights to soften the behavior without retraining the model. Do not describe how neurons are scored or selected.
- Contrastive demographic prompt pairs. Explain how two otherwise identical prompts isolate a demographic attribute for comparison, using the white-versus-Black pedestrian example. Describe why differences in the model’s responses can then be attributed to that one changed attribute.
- Equal-length prompt verification. Explain how to check that a contrastive prompt pair has the same number of tokens and why that equality matters for comparing neuron activations.
- GLU activation capture point. Explain why down_proj_input is used to inspect neuron responses: up_proj and gate_proj have already expanded and combined the information, but down_proj has not yet contracted it.
- Targeted neuron candidate selection. Explain what TOP_K selects and why the chapter aims to modify as few neurons as possible. State that five is a tunable choice for this model and prompt pair, not a universal rule.
- Deterministic baseline generation. Explain why the section checks both prompt completions before measuring neuron contributions and how greedy decoding makes later output comparisons repeatable.
- Contrasting pedestrian completions. Describe how the model completed the two pedestrian prompts and explain the significance of changing only the demographic variable.
- Per-neuron activation differences. Explain how the section reduces each prompt’s activations to one mean value per neuron and computes MagDiff for each layer.
- Activation magnitude and polarity. Distinguish a change in activation magnitude from a change in polarity using the two prompts’ separately recorded activation signs.
- Difference heatmap construction. Describe how the section turns per-neuron magnitude differences into a heatmap across layers.
- Layerwise difference pattern. Explain what the heatmap shows about how prompt differences vary across the model's layers.
- Global neuron score list. Describe how per-layer signed scores become one list in which each neuron can be identified and compared across layers.
- Magnitude based candidate ranking. Explain how top-k candidate neurons are selected from the flat list and why the direction of a difference does not affect their rank.
- Up projection scaling effect. Explain why scaling a neuron's up_proj contribution reduces its contribution to down_proj_input by the same factor.
- Per neuron weight intervention. Describe how apply_explicit_scales uses EXPLICIT_SCALES to reduce the five selected neurons' contributions.
- Proportional activation change. Explain why scaling the five selected neurons to one tenth of their original weight made their measured MagDiff approximately ten times smaller.
- Observed generation contrast. Describe how the generated text changed on the stereotyped and majority sides after scaling, and state the limit of this observation.
- Cross-scenario control test. Explain why the five neurons were tested with a new prompt pair and how the comparison was set up.
- Job-scenario response contrast. Describe the contrast between the baseline responses and what changed after the intervention in the job scenario.
- Non-directional scaling effect. Explain why the shared post-intervention outcome should not be read as a deliberately chosen answer.
- Gender contrastive prompt setup. Describe the gender-contrastive prompt pair: both prompts have 19 tokens, with the “boy” prompt designated the majority prompt and the “girl” prompt designated the stereotyped prompt.
- Gender candidate neuron locations. Describe where the top five gender-bias candidate neurons appear and how that pattern compares with the racial-bias candidates. Explain why no overlap between the two top-five lists is inconclusive.
- Single-neuron scaling setup. Describe how the intervention scaled the selected neuron in layer 10 by 0.4 while leaving the other four top-k neurons at 1, so only one entry changed the model weights.
- Uneven response change. Explain what changed in the paired responses after the single-neuron intervention and why the result is a softening of the difference, not a complete resolution.
- Layerwise effects of neuron scaling. Trace how scaling L10:5982 affects predictions from layers 10 to 15 for the majority and stereotyped prompts. Explain why the intervention's effect is visible in later layers rather than at L10 itself.
- Complementary intervention checks. Explain why responses to specific prompts are not enough to evaluate the two intervened models: changing weights may affect general capabilities, and individual generations do not quantify bias. Describe the roles of comparison with the base model on general benchmarks and a bias-specific benchmark such as BBQ.
- General capability benchmark results. Explain how the section checks whether the race and gender interventions harmed general capabilities, and describe the result. Relate that result to the small number of neurons whose weights were scaled.
- BBQ question contexts. Explain how BBQ uses ambiguous and disambiguated versions of a question to test whether a model follows the available information rather than a social stereotype.
- Bias score interpretation. Explain what the sign and proximity to zero of a BBQ bias score mean, and distinguish what the scores test in ambiguous and disambiguated contexts.
- Intervention benchmark results. Describe how the base and intervened models' race/ethnicity bias scores compare, and explain what that comparison supports.
- Resource gap and simpler methods. Explain why the scale of frontier-model research motivates simpler methods for studying and intervening in smaller language models.
- Superposition and sparse features. Explain why an MLP neuron can respond to multiple topics and how a sparse autoencoder helps isolate more interpretable features.
- Feature steering and its costs. Describe how Anthropic steers a located concept feature and why both finding and controlling such features can be costly.
- Built-in expansion and permanent edits. Distinguish the chapter’s neuron-weight approach from SAE-based steering, including its trade-off.
- Category-specific candidate neurons. Explain what the paper’s overlap measurement says about whether candidate bias neurons are shared across demographic categories.
- Candidate selection and intervention. Distinguish the candidate-selection method shared by the paper and chapter from their different interventions, and explain what scaling permits.
- Single-pair evidence and limits. Explain what the chapter’s single-pair experiment showed and how its evidential scale differs from the paper’s.
What it builds on
- Capturing activations in transformers (chapter 10) · Capturing activations in transformers.
- The GLU mechanism in action (chapter 10) · The GLU mechanism within the MLP.
- The evolution of the MLP: from simple expansion to Gated Linear Units (GLU) (chapter 3) · The MLP’s GLU structure and expansion.
- From noise to answer across the layers (chapter 10) · The earlier introduction of the logit lens for inspecting predictions across layers.
- Mapping neuron activity across the model (chapter 10) · Observation of neuron activity within the model
Section by section
Introduction
The introduction defines demographic bias, explains the basic idea of comparing prompts and adjusting a few neurons, and clarifies the limits of that technique.
- Demographic bias in model responses: Explain that this chapter calls it bias when a model treats an otherwise identical case differently because a demographic attribute such as race or gender changes, and that internet-derived training data can contribute to unwanted responses.
Why it matters: This definition identifies the specific behavior the chapter aims to examine. - Contrastive comparison and targeted adjustment: Describe the chapter's basic approach: compare two prompts that differ only in a demographic variable to observe differing neuron activations, then adjust a few neurons' weights to soften the behavior without retraining the model. Do not describe how neurons are scored or selected.
Why it matters: It connects the controlled comparison to the chapter's proposed localized intervention. - Limits of technical bias intervention: Distinguish a technique for changing how a model processes a demographic attribute from deciding which behavior counts as bias or deserves correction; note that the model's predominantly English-language, American-influenced training data shapes its learned associations.
Why it matters: It keeps a model-editing method from being mistaken for a complete fairness decision.
Key terms: bias, demographic attribute, contrastive prompt pairs, activations, neurons, associations
11.1 Comparing activations between prompts
The section introduces contrastive prompt pairs as a way to isolate a demographic attribute when comparing model responses.
- Contrastive demographic prompt pairs: Explain how two otherwise identical prompts isolate a demographic attribute for comparison, using the white-versus-Black pedestrian example. Describe why differences in the model’s responses can then be attributed to that one changed attribute.
Why it matters: Keeping the prompts equivalent helps you tell whether the demographic change is associated with a change in the model’s response.
Key terms: contrastive prompt, demographic variable, fully equivalent prompts
11.1.1 Creating and verifying the prompt pairs
This section shows how to verify that two prompts have equal token lengths so their neuron activations can be compared at corresponding positions.
- Equal-length prompt verification: Explain how to check that a contrastive prompt pair has the same number of tokens and why that equality matters for comparing neuron activations.
Why it matters: Without equal token counts, the same position in the two prompts may refer to different points in the text.
Key terms: tokenizer, token identifiers, tensor, n_tokens, assert #1
11.1.2 Where the differences show up
This section explains why the input to the MLP’s down_proj is the capture point for comparing neuron responses to demographic changes in prompts.
- GLU activation capture point: Explain why down_proj_input is used to inspect neuron responses: up_proj and gate_proj have already expanded and combined the information, but down_proj has not yet contracted it.
Why it matters: The capture point determines which neuron responses can be compared between prompts. - Expanded layer targeting rationale: Justify focusing on the MLP’s expanded layers when looking for neurons that respond differently to demographic variables.
Why it matters: It explains why this location is preferable when trying to limit effects beyond the targeted behavior.
Key terms: up_proj, gate_proj, down_proj, down_proj_input, GLU mechanism, capture point, expanded layers, neurons, specialized, other topics
11.2 Finding and scaling down the neurons behind racial bias
This section sets up a targeted neuron intervention and establishes that changing only the pedestrian’s race in a prompt produces markedly different baseline completions.
- Targeted neuron candidate selection: Explain what TOP_K selects and why the chapter aims to modify as few neurons as possible. State that five is a tunable choice for this model and prompt pair, not a universal rule.
Why it matters: It explains why the proposed intervention focuses on a tiny, adjustable set of candidates rather than changing neurons broadly. - Deterministic baseline generation: Explain why the section checks both prompt completions before measuring neuron contributions and how greedy decoding makes later output comparisons repeatable.
Why it matters: A repeatable baseline makes a later comparison with an intervened model interpretable. - Contrasting pedestrian completions: Describe how the model completed the two pedestrian prompts and explain the significance of changing only the demographic variable.
Why it matters: The contrasting completions establish the response difference that motivates investigating candidate neurons.
Key terms: TOP_K, activation difference, candidate neurons, MLP module, do_sample=False, greedy decoding, baseline, deterministic generation, majority, stereotyped prompt
11.2.1 Measuring the activation difference
This section computes per-neuron activation differences between a majority prompt and a stereotyped prompt, then uses activation signs to distinguish changes in magnitude from changes in polarity.
- Per-neuron activation differences: Explain how the section reduces each prompt’s activations to one mean value per neuron and computes MagDiff for each layer.
Why it matters: This comparison gives a neuron-by-neuron measure of how responses to the two prompts differ. - Activation magnitude and polarity: Distinguish a change in activation magnitude from a change in polarity using the two prompts’ separately recorded activation signs.
Why it matters: Checking signs prevents a simple shift in activation from being mistaken for a reversal in direction.
Key terms: MagDiff, magnitude_diff, get_prompt_activations, signed_scores, maj_sign, ster_sign, torch.sign(), polarity
11.2.2 Visualizing the differences across layers
This section builds a binned heatmap of neuron activation differences between prompts, interprets where those differences are strongest, and explains why the heatmap cannot identify individual neurons for intervention.
- Difference heatmap construction: Describe how the section turns per-neuron magnitude differences into a heatmap across layers.
Why it matters: The construction determines what the heatmap can show about differences across the model. - Layerwise difference pattern: Explain what the heatmap shows about how prompt differences vary across the model's layers.
Why it matters: Reading the pattern focuses attention on the layers with the most visible prompt-related variation. - Binned heatmap limitation: Distinguish locating a region of activation differences from identifying individual neurons to act on.
Why it matters: A visually striking bin is not itself a precise neuron-selection result.
Key terms: MagDiff, compute_signed_bias, magnitude_diff, BIN_SIZE, bin_neurons, plot_heatmap_panel, L0 to L2, L3, L15, activation differences
11.2.3 Selecting the top candidate neurons
This section turns per-layer activation differences into a global ranking of candidate neurons and examines the five highest-ranked candidates.
- Global neuron score list: Describe how per-layer signed scores become one list in which each neuron can be identified and compared across layers.
Why it matters: A flat list makes neurons from different layers directly comparable. - Magnitude based candidate ranking: Explain how top-k candidate neurons are selected from the flat list and why the direction of a difference does not affect their rank.
Why it matters: Ranking by size finds the largest changes without overlooking changes in the negative direction. - Top five neuron pattern: Describe where the five selected neurons occur and what their prompt-sign comparison shows.
Why it matters: The selected candidates reflect the late-layer concentration seen in the heatmap, while their differences are changes in size rather than sign reversals.
Key terms: signed_scores, flatten_signed_scores, magnitude_diff, maj_sign, ster_sign, flat_scores, topk_signed, top_k, Maj_Dir, Ster_Dir
11.2.4 Intervening on the top-k neurons
This section explains how to reduce the contributions of five selected neurons by scaling their up_proj weights, after restoring the model to its original weights.
- Up projection scaling effect: Explain why scaling a neuron's up_proj contribution reduces its contribution to down_proj_input by the same factor.
Why it matters: It explains why changing one projection gives a predictable way to reduce a neuron's contribution. - Per neuron weight intervention: Describe how apply_explicit_scales uses EXPLICIT_SCALES to reduce the five selected neurons' contributions.
Why it matters: The dictionary and row-level change target selected neurons rather than scaling the whole model. - Original weight restoration: Explain why restore_model is called before applying the neuron scales.
Why it matters: Restoration prevents earlier notebook executions from changing the starting point of this intervention.
Key terms: up_proj, gate_proj, down_proj_input, SiLU, scale, apply_explicit_scales, EXPLICIT_SCALES, top-k neurons, MLP, restore_model
11.2.5 What changed after scaling
The section checks how scaling five selected neurons changed their activation differences and the model’s responses to one prompt pair.
- Proportional activation change: Explain why scaling the five selected neurons to one tenth of their original weight made their measured MagDiff approximately ten times smaller.
Why it matters: It connects the weight intervention to the measured change in neuron behavior. - Observed generation contrast: Describe how the generated text changed on the stereotyped and majority sides after scaling, and state the limit of this observation.
Why it matters: The comparison shows a substantial output change from a very small intervention, without establishing that it generalizes.
Key terms: MagDiff, up_proj, down_proj_input, SiLU, EXPLICIT_SCALES, stereotyped side, majority side, top-k neurons, GLU neurons
11.2.6 Same bias, different scenario
The section tests whether scaling down five selected neurons changes racial-attribute differences in a new scenario, then interprets what the resulting responses do and do not show.
- Cross-scenario control test: Explain why the five neurons were tested with a new prompt pair and how the comparison was set up.
Why it matters: A changed response in a second context is stronger evidence than a result from the prompt pair used to select the neurons. - Job-scenario response contrast: Describe the contrast between the baseline responses and what changed after the intervention in the job scenario.
Why it matters: The before-and-after responses are the section’s evidence that the intervention also affected a different scenario on the same racial axis. - Non-directional scaling effect: Explain why the shared post-intervention outcome should not be read as a deliberately chosen answer.
Why it matters: Matching outputs show a change in this test, not that the intervention was designed to make both stories favor police work.
Key terms: prompt pair, racial attribute, five selected neurons, same scaling, baseline responses, intervention, majority version, stereotyped version, scaling down, activation difference
11.3 Replicating the procedure with gender bias
The section introduces the gender-bias experiment by setting up a token-length-matched pair of contrastive prompts.
- Gender contrastive prompt setup: Describe the gender-contrastive prompt pair: both prompts have 19 tokens, with the “boy” prompt designated the majority prompt and the “girl” prompt designated the stereotyped prompt.
Why it matters: This pair is the starting point for the gender-bias experiment.
Key terms: contrastive prompts, majority prompt, stereotyped prompt
11.3.1 Locating the gender neurons
The section locates the top five gender-bias candidate neurons, compares their positions with the racial-bias candidates, and cautions against overinterpreting the lack of overlap.
- Gender candidate neuron locations: Describe where the top five gender-bias candidate neurons appear and how that pattern compares with the racial-bias candidates. Explain why no overlap between the two top-five lists is inconclusive.
Why it matters: The comparison identifies candidate locations without treating a small top-five sample as proof of separate bias circuits. - Layer ten intervention reach: State what the section says about the effect of intervening on the candidate neuron in layer 10.
Why it matters: A candidate’s location matters because an intervention there can affect later layers.
Key terms: Top 5 neuron list, deepest blocks, MLP module, layer 10, intervention
11.3.2 Not every neuron has the same influence
This subsection tests whether scaling one selected gender-related neuron can change a stereotyped response without changing the paired majority-side response.
- Single-neuron scaling setup: Describe how the intervention scaled the selected neuron in layer 10 by 0.4 while leaving the other four top-k neurons at 1, so only one entry changed the model weights.
Why it matters: The setup isolates the effect of scaling one selected neuron rather than all five candidates. - Uneven response change: Explain what changed in the paired responses after the single-neuron intervention and why the result is a softening of the difference, not a complete resolution.
Why it matters: The observed change shows a targeted effect while making clear that one altered response is not a complete fix.
Key terms: layer 10, factor of 0.4, factor of 1, EXPLICIT_SCALES_GENDER, top-k, stereotyped response, majority-side response, age condition, pronouns
11.3.3 Tracing the effect across layers
The section uses a logit lens to show how scaling a gender-bias-associated neuron affects predictions across later layers while leaving the majority prompt largely unchanged.
- Layerwise effects of neuron scaling: Trace how scaling L10:5982 affects predictions from layers 10 to 15 for the majority and stereotyped prompts. Explain why the intervention's effect is visible in later layers rather than at L10 itself.
Why it matters: Looking across layers shows both where the intervention changes predictions and where it leaves them alone. - Logit lens and generated words: Distinguish the logit lens predictions from the words selected during generation in this comparison.
Why it matters: This distinction prevents a difference between the figure and generated text from being mistaken for an inconsistency in the analysis.
Key terms: logit lens, L10:5982, majority prompt, stereotyped prompt, repetition_penalty=1.1, post-hoc adjustments
11.4 Benchmarking the intervened models
This section explains why targeted weight interventions need both general-capability and bias-specific evaluation beyond inspecting responses to individual prompts.
- Complementary intervention checks: Explain why responses to specific prompts are not enough to evaluate the two intervened models: changing weights may affect general capabilities, and individual generations do not quantify bias. Describe the roles of comparison with the base model on general benchmarks and a bias-specific benchmark such as BBQ.
Why it matters: A promising response to a prompt does not establish either that general capabilities were preserved or that bias changed measurably.
Key terms: intervened models, base model, general benchmarks, BBQ, bias-specific benchmark
11.4.1 Measuring the impact on general capabilities
The section compares the base model with race- and gender-intervened versions on general-capability benchmarks and reports no measurable loss.
- General capability benchmark results: Explain how the section checks whether the race and gender interventions harmed general capabilities, and describe the result. Relate that result to the small number of neurons whose weights were scaled.
Why it matters: This comparison checks whether the targeted interventions came at a measurable cost to the model’s general capabilities.
Key terms: base model, race-intervened model, gender-intervened model, general capabilities, MLP module
11.4.2 Measuring bias with BBQ
This section uses BBQ to test whether a five-neuron racial-bias intervention changes behavior beyond the prompts used to select the neurons.
- BBQ question contexts: Explain how BBQ uses ambiguous and disambiguated versions of a question to test whether a model follows the available information rather than a social stereotype.
Why it matters: The two contexts reveal different ways a stereotype can displace an evidence-based answer. - Bias score interpretation: Explain what the sign and proximity to zero of a BBQ bias score mean, and distinguish what the scores test in ambiguous and disambiguated contexts.
Why it matters: A score moving toward zero has a different meaning from simply answering more questions correctly. - BBQ evaluation setup: Describe how the section runs BBQ on the intervened model and which results it extracts.
Why it matters: The setup connects the intervention tested on individual prompts to a broader benchmark comparison. - Intervention benchmark results: Describe how the base and intervened models' race/ethnicity bias scores compare, and explain what that comparison supports.
Why it matters: The benchmark checks whether a prompt-level improvement reflects a broader change in the targeted behavior.
Key terms: BBQ, ambiguous, disambiguated, not enough information, BBQ bias score, ambiguous context, disambiguated context, stereotype, run_bbq, HFLM
11.5 From paper to practice
This section explains why the resource demands of frontier-model interpretability research create a need for simpler, affordable precision-intervention methods.
- Resource gap and simpler methods: Explain why the scale of frontier-model research motivates simpler methods for studying and intervening in smaller language models.
Why it matters: It explains why a practical, lower-cost approach is worth considering alongside frontier-scale research.
Key terms: Anthropic, SLMs, Fairness Pruning
11.5.1 Anthropic, the starting point
The section contrasts Anthropic’s feature-based interpretability and steering with the chapter’s less costly, permanent neuron-weight intervention.
- Superposition and sparse features: Explain why an MLP neuron can respond to multiple topics and how a sparse autoencoder helps isolate more interpretable features.
Why it matters: This explains why locating a behavior in ordinary neurons is harder than locating an interpretable feature. - Feature steering and its costs: Describe how Anthropic steers a located concept feature and why both finding and controlling such features can be costly.
Why it matters: It distinguishes a temporary inference-time intervention from a change built into the model. - Built-in expansion and permanent edits: Distinguish the chapter’s neuron-weight approach from SAE-based steering, including its trade-off.
Why it matters: It explains both the practical appeal of permanent edits and the need to check their wider effects.
Key terms: superposition, sparse autoencoder (SAE), features, polysemantic neurons, steering, clamping, inference-time control, SAE, up_proj, gate_proj
11.5.2 Fairness Pruning, one step ahead
Connects the chapter’s neuron-pruning experiments to Fairness Pruning’s evidence, methods, and limits.
- Bias signal location: Describe where the paper found the strongest demographic-attribute signal across its tested models, languages, and categories, and how gender differed.
Why it matters: It puts the chapter’s race and gender neuron findings in the context of the paper’s broader results. - Category-specific candidate neurons: Explain what the paper’s overlap measurement says about whether candidate bias neurons are shared across demographic categories.
Why it matters: Limited overlap helps explain why neurons selected for one bias need not be the neurons selected for another. - Candidate selection and intervention: Distinguish the candidate-selection method shared by the paper and chapter from their different interventions, and explain what scaling permits.
Why it matters: It separates evidence used to find candidate neurons from the intervention used to test and adjust their influence. - Single-pair evidence and limits: Explain what the chapter’s single-pair experiment showed and how its evidential scale differs from the paper’s.
Why it matters: It lets you describe the promising transfer result without overstating what one prompt pair and one model establish. - Behavior beyond fairness: State the condition under which the section says the procedure could apply beyond bias.
Why it matters: It identifies the stated condition for considering the same approach for a different behavior.
Key terms: last layer, gender, signal, Jaccard index, BiasScore, candidate neurons, demographic categories, contrastive prompt pairs, down_proj, up_proj
11.6 Hands-on lab
11.7 Summary
Concepts explained in this chapter
- activation differenceA signed activation difference is the stereotyped-prompt mean activation minus the majority-prompt mean activation for a neuron.
- BBQBBQ (Bias Benchmark for QA) is a question-answering benchmark dataset designed to measure social biases in language models.
- BBQ bias scoreA BBQ bias score measures the extent to which a model's answers align with stereotypes rather than contradict them.
- contrastive prompt pairA paired-prompt method compares prompts that differ in one controlled attribute to isolate its effect on model behavior.
- sparse autoencoderA sparse autoencoder is an autoencoder that reconstructs neural activations through a wider intermediate representation constrained to activate sparsely.
- fairness pruningFairness pruning is a model-modification technique that removes selected neurons to change unwanted behavioral tendencies while preserving performance on desired capabilities.
- tokenizationTokenization is the process of converting text into discrete numerical units that a language model can process.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
