Distance measures how far apart objects or representations are. In natural language processing, character-based measures can miss meaning: “good” and “okay” differ in spelling, while “bad” and “bag” look alike but mean different things. Vector-based distances are useful when comparing word, text, topic, or document representations. section 1.5.3 section 4.6
Common choices include:
- Euclidean (L2), squared Euclidean, cosine, Minkowski, Manhattan (L1), Jaccard, Mahalanobis, and Levenshtein (edit) distance. section 4.6
- Choose a measure that suits the representation and task: edit distance counts character changes, while vector distances can compare represented text. section 1.5.3
Checking topic representations
Compare distances between LSA topic vectors with those from a higher-dimensional TF–IDF representation. LSA preserves large distances but may not retain close relationships, so nearby topic vectors do not always reflect fine-grained similarity in document subject matter. section 4.6
Listing 4.12 Pairwise distances available in sklearn
'cityblock', 'cosine', 'euclidean', 'l1', 'l2', 'manhattan', 'braycurtis', 'canberra', 'chebyshev', 'correlation', 'dice', 'hamming', 'jaccard', 'kulsinski', 'mahalanobis', 'matching', 'minkowski', 'rogerstanimoto', 'russellrao', 'seuclidean', 'sokalmichener', 'sokalsneath', 'sqeuclidean', 'yule'
Browse the pairwise distance options listed for scikit-learn. section 4.6
