A similarity metric gives a numerical score for how closely two texts match. This approach scores identical documents as 1 and documents with no words in common as 0, using word counts rather than word order. section 29.0 section 29.1
Calculate the score through a small sequence of reusable tasks:
- Read each file and turn its contents into a list of words. section 29.1
- Build a frequency dictionary that maps every word to its number of occurrences. section 29.4
- Add the absolute differences between matching word counts and the counts of words appearing in only one document. section 29.5
- Divide the difference total by the combined word count, then subtract the ratio from 1. section 29.5
Testing the metric
Compare a file with itself to check for 1.0, then compare a sonnet with an empty file to check for 0.0. Shakespeare’s “Sonnet 18” and “Sonnet 19” produce 0.24, while changing three instances of “summer” to “winter” in “Sonnet 18” produces 0.97. section 29.6
Listing 29.3. Making a frequency dictionary for words
def frequencies(words):
"""
words: list of words
returns: frequency dictionary for input words
"""
freq_dict = {} #1
for word in words: #2
if word in freq_dict: #3
freq_dict[word] += 1 #4
else: #5
freq_dict[word] = 1 #6
return freq_dict #7
freq_dict = frequencies(words) #8
Inspect the frequency-dictionary function that counts each word with a Python dictionary. section 29.4
Listing 29.4. Calculate similarity given two input dictionaries
def calculate_similarity(dict1, dict2):
"""
dict1: frequency dictionary for one text
dict2: frequency dictionary for another text
returns: float, representing how similar both texts are to each other
"""
diff = 0
total = 0
for word in dict1.keys(): #1
if word in dict2.keys(): #2
diff += abs(dict1[word] - dict2[word]) #3
else: #4
diff += dict1[word] #5
for word in dict2.keys(): #6
if word not in dict1.keys(): #7
diff += dict2[word] #8
total = sum(dict1.values()) + sum(dict2.values()) #9
difference = diff / total #10
similar = 1.0 – difference #11
return round(similar, 2) #12
Inspect the comparison function that accumulates word-count differences and returns a similarity value. section 29.5
