Jaccard similarity measures overlap between two sets: divide the size of their intersection by the size of their union. Here, it compares the top-magnitude activation channels in consecutive transformer layers. section 7.1
Comparing channel overlap across models
Consecutive layers in OPT-6.7B have a Jaccard similarity of 0.70 for top channels, compared with 0.54 in OPT-125M. section 7.1
Figure 7.1 Per-channel activation magnitude heatmap for OPT-6.7B (32 layers × 4,096 hidden dimensions, log scale). Bright vertical stripes mark the same hidden dimensions spiking across all layers. Dashed lines mark the five universal outlier channels—dimensions 639, 1778, 2394, 2972, and 3136—that appear in the top 1% of every layer.

The heatmap shows hidden dimensions with high activation magnitudes recurring across OPT-6.7B layers. section 7.1


