liveBook · Manning

Jaccard similarity

As explained in Quantization and Fast Inference

Jaccard similarity is the size of the intersection of two sets divided by the size of their union.

Jaccard similarity measures overlap between two sets: divide the size of their intersection by the size of their union. Here, it compares the top-magnitude activation channels in consecutive transformer layers. section 7.1

Comparing channel overlap across models

Consecutive layers in OPT-6.7B have a Jaccard similarity of 0.70 for top channels, compared with 0.54 in OPT-125M. section 7.1

Figure 7.1 Per-channel activation magnitude heatmap for OPT-6.7B (32 layers × 4,096 hidden dimensions, log scale). Bright vertical stripes mark the same hidden dimensions spiking across all layers. Dashed lines mark the five universal outlier channels—dimensions 639, 1778, 2394, 2972, and 3136—that appear in the top 1% of every layer.

The heatmap shows hidden dimensions with high activation magnitudes recurring across OPT-6.7B layers. section 7.1

How Quantization and Fast Inference covers it

How deep it goes
Defines the intersection-over-union measure and applies it to adjacent layers, without deriving it further.
Examples
Contrasts consecutive-layer similarities of 0.70 for OPT-6.7B and 0.54 for OPT-125M.
What you can do after
Evaluate whether high-magnitude channels recur across adjacent layers using set overlap.
How it connects
Connects set intersection and union to activation outliers and quantization decisions.

Also explained in

Compare how 3 books explain Jaccard similarity →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Quantization and Fast Inference. The section links point to the parts of the book it draws on.