liveBook · Manning

Chapter 11: Precision pruning for bias

What you will learn in chapter 11 of Rearchitecting LLMs

The introduction defines demographic bias, explains the basic idea of comparing prompts and adjusting a few neurons, and clarifies the limits of that technique.

After this chapter you will be able to

What it builds on

Section by section

Introduction

The introduction defines demographic bias, explains the basic idea of comparing prompts and adjusting a few neurons, and clarifies the limits of that technique.

Key terms: bias, demographic attribute, contrastive prompt pairs, activations, neurons, associations

11.1 Comparing activations between prompts

The section introduces contrastive prompt pairs as a way to isolate a demographic attribute when comparing model responses.

Key terms: contrastive prompt, demographic variable, fully equivalent prompts

11.1.1 Creating and verifying the prompt pairs

This section shows how to verify that two prompts have equal token lengths so their neuron activations can be compared at corresponding positions.

Key terms: tokenizer, token identifiers, tensor, n_tokens, assert #1

11.1.2 Where the differences show up

This section explains why the input to the MLP’s down_proj is the capture point for comparing neuron responses to demographic changes in prompts.

Key terms: up_proj, gate_proj, down_proj, down_proj_input, GLU mechanism, capture point, expanded layers, neurons, specialized, other topics

11.2 Finding and scaling down the neurons behind racial bias

This section sets up a targeted neuron intervention and establishes that changing only the pedestrian’s race in a prompt produces markedly different baseline completions.

Key terms: TOP_K, activation difference, candidate neurons, MLP module, do_sample=False, greedy decoding, baseline, deterministic generation, majority, stereotyped prompt

11.2.1 Measuring the activation difference

This section computes per-neuron activation differences between a majority prompt and a stereotyped prompt, then uses activation signs to distinguish changes in magnitude from changes in polarity.

Key terms: MagDiff, magnitude_diff, get_prompt_activations, signed_scores, maj_sign, ster_sign, torch.sign(), polarity

11.2.2 Visualizing the differences across layers

This section builds a binned heatmap of neuron activation differences between prompts, interprets where those differences are strongest, and explains why the heatmap cannot identify individual neurons for intervention.

Key terms: MagDiff, compute_signed_bias, magnitude_diff, BIN_SIZE, bin_neurons, plot_heatmap_panel, L0 to L2, L3, L15, activation differences

11.2.3 Selecting the top candidate neurons

This section turns per-layer activation differences into a global ranking of candidate neurons and examines the five highest-ranked candidates.

Key terms: signed_scores, flatten_signed_scores, magnitude_diff, maj_sign, ster_sign, flat_scores, topk_signed, top_k, Maj_Dir, Ster_Dir

11.2.4 Intervening on the top-k neurons

This section explains how to reduce the contributions of five selected neurons by scaling their up_proj weights, after restoring the model to its original weights.

Key terms: up_proj, gate_proj, down_proj_input, SiLU, scale, apply_explicit_scales, EXPLICIT_SCALES, top-k neurons, MLP, restore_model

11.2.5 What changed after scaling

The section checks how scaling five selected neurons changed their activation differences and the model’s responses to one prompt pair.

Key terms: MagDiff, up_proj, down_proj_input, SiLU, EXPLICIT_SCALES, stereotyped side, majority side, top-k neurons, GLU neurons

11.2.6 Same bias, different scenario

The section tests whether scaling down five selected neurons changes racial-attribute differences in a new scenario, then interprets what the resulting responses do and do not show.

Key terms: prompt pair, racial attribute, five selected neurons, same scaling, baseline responses, intervention, majority version, stereotyped version, scaling down, activation difference

11.3 Replicating the procedure with gender bias

The section introduces the gender-bias experiment by setting up a token-length-matched pair of contrastive prompts.

Key terms: contrastive prompts, majority prompt, stereotyped prompt

11.3.1 Locating the gender neurons

The section locates the top five gender-bias candidate neurons, compares their positions with the racial-bias candidates, and cautions against overinterpreting the lack of overlap.

Key terms: Top 5 neuron list, deepest blocks, MLP module, layer 10, intervention

11.3.2 Not every neuron has the same influence

This subsection tests whether scaling one selected gender-related neuron can change a stereotyped response without changing the paired majority-side response.

Key terms: layer 10, factor of 0.4, factor of 1, EXPLICIT_SCALES_GENDER, top-k, stereotyped response, majority-side response, age condition, pronouns

11.3.3 Tracing the effect across layers

The section uses a logit lens to show how scaling a gender-bias-associated neuron affects predictions across later layers while leaving the majority prompt largely unchanged.

Key terms: logit lens, L10:5982, majority prompt, stereotyped prompt, repetition_penalty=1.1, post-hoc adjustments

11.4 Benchmarking the intervened models

This section explains why targeted weight interventions need both general-capability and bias-specific evaluation beyond inspecting responses to individual prompts.

Key terms: intervened models, base model, general benchmarks, BBQ, bias-specific benchmark

11.4.1 Measuring the impact on general capabilities

The section compares the base model with race- and gender-intervened versions on general-capability benchmarks and reports no measurable loss.

Key terms: base model, race-intervened model, gender-intervened model, general capabilities, MLP module

11.4.2 Measuring bias with BBQ

This section uses BBQ to test whether a five-neuron racial-bias intervention changes behavior beyond the prompts used to select the neurons.

Key terms: BBQ, ambiguous, disambiguated, not enough information, BBQ bias score, ambiguous context, disambiguated context, stereotype, run_bbq, HFLM

11.5 From paper to practice

This section explains why the resource demands of frontier-model interpretability research create a need for simpler, affordable precision-intervention methods.

Key terms: Anthropic, SLMs, Fairness Pruning

11.5.1 Anthropic, the starting point

The section contrasts Anthropic’s feature-based interpretability and steering with the chapter’s less costly, permanent neuron-weight intervention.

Key terms: superposition, sparse autoencoder (SAE), features, polysemantic neurons, steering, clamping, inference-time control, SAE, up_proj, gate_proj

11.5.2 Fairness Pruning, one step ahead

Connects the chapter’s neuron-pruning experiments to Fairness Pruning’s evidence, methods, and limits.

Key terms: last layer, gender, signal, Jaccard index, BiasScore, candidate neurons, demographic categories, contrastive prompt pairs, down_proj, up_proj

11.6 Hands-on lab

11.7 Summary

Concepts explained in this chapter

← Chapter 10: Exploring the transformer black box Chapter 12: Case study — replacing one agent call with a specialized SLM in production →

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.