concept similarity in category data

appears as: similarity, similarity, similarities, The similarity, similarities
Data Science Bookcamp: Five Python Projects

This is an excerpt from Manning's book Data Science Bookcamp: Five Python Projects.

We have successfully clustered our headlines by location, and plotted those clusters where the word Zika is dominant. This relationship between our clusters and their textual content leads to an interesting question; is it possible to cluster the headlines based on text similarity rather than geographic distance? In other words, can we group our headlines by text overlap, so that all the references to Zika automatically appear in a single cluster? Yes we can! In the subsequent Case Study, we learn how to measure similarity between texts in order to group documents by topic.

  • The similarity is simple to implement.
  • Listing 13.10. Efficiently computing the Jaccard similarity
    def jaccard_similarity_efficient(text_a, text_b):
        word_set_a, word_set_b = [set(simplify_text(text).split())
                                  for text in [text_a, text_b]]
        num_shared = len(word_set_a & word_set_b)
        num_total = len(word_set_a) + len(word_set_b) - num_shared #1
        return num_shared / num_total
    
    for text in [text2, text3]:
        similarity = jaccard_similarity_efficient(text1, text)
        assert similarity == jaccard_similarity(text1, text)
    Figure 13.13. A table of normalized of Tanimoto similarities across text-pairs. The table’s diagonal represents the similarity between each text and itself. Not surprisingly, that similarity is 1. Ignoring the diagonal, we see that texts 1 and 2 share the highest similarity. Meanwhile, texts 2 and 3 share the lowest similarity.
    fig13 13