Semantic similarity measures whether language items express related meanings, even when they use different words. Dense embeddings make those relationships searchable, unlike conventional full-text matching. section 10.3.2
A transformer converts sentences into vectors; comparing normalized vectors with a dot product produces cosine similarity. Similarity values can form a matrix or heatmap for inspecting related passages and possible redundancy. section 11.2.3
Finding related passages
Encode 672 body-text passages with all-MiniLM-L12-v2 into 384-dimensional vectors, normalize them, compute every pairwise cosine similarity, and visualize the resulting 672-by-672 matrix. Values above roughly 0.85 can flag statements that may be redundant, but the heatmap still requires interpretation. section 11.2.3
>>> texts = df9.text[df9.is_body]
>>> texts.shape
(672,)
>>> from sentence_transformers import SentenceTransformer
>>> minibert = SentenceTransformer('all-MiniLM-L12-v2')
>>> vecs = minibert.encode(list(texts))
>>> vecs.shape
(672, 384)
Inspect the executable embedding step that turns selected body-text passages into 384-dimensional vectors. section 11.2.3
