An activation is a value produced by a neuron as it processes an input. Unlike learned weights, which are static parameters at inference time, activations vary with the input and reveal how the model's components respond. A layer produces an activation vector that becomes input to subsequent computation. section 3.3 section 3.5
Figure 3.9 How activations flow between layers. Each neuron applies its static weights (learned during training) to the input values and passes the result through an activation function (SiLU). The output values, the activations, vary in intensity: high values propagate strong signals, while low values suppress information. These activations become the inputs for the next layer, creating the continuous flow of information through the model.
Follow how weighted inputs pass through a function and become activations for the next layer. section 3.3
Capturing activations makes it possible to inspect which neurons respond to particular tokens and how their responses change across layers. This supports explainability and data-driven optimization: activation patterns can guide neuron selection for width pruning or help locate behavior to investigate before fairness pruning. section 1.1.3 section 5.0 section 1.4
A practical inspection workflow uses captured tensors rather than treating a Transformer block as opaque:
- Choose a block or submodule and attach PyTorch forward hooks or forward pre-hooks to capture its outputs or inputs. section 10.0 section 10.6
- Check the captured tensor's shape before plotting. Group neurons into bins and plot each bin's mean activation as a heatmap to compare tokens and layers. section 10.2
- Inspect the GLU components side by side to see how multiplication filters the signal. Use a logit lens separately to project intermediate hidden states through final normalization and the prediction head. section 10.6
- Verify that hooks actually capture data when changing model families. Attention submodule names can differ, so a hook can register without capturing the intended activations. section 10.5
Inspecting a four-token tensor
In Llama-3.2-1B, a single-layer representation of four tokens contains four vectors of 2,048 values. An expanded gate_proj vector has 8,192 values. Binning these values and coloring their means makes the activation pattern manageable to inspect, though the plot summarizes groups rather than showing every neuron separately. section 10.2
For bias analysis, compare activation snapshots from prompt pairs that differ in demographic characteristics. The differences provide a signal for locating possible bias paths; they are a starting point for investigation and intervention, not a performance result by themselves. section 10.0





