The different-to-total ratio measures word variety by dividing the number of distinct words by the total number of words. It helps form an authorship signature alongside other text features. section 11.3.1
A short sample
“A pearl! Pearl! Lustrous pearl! Rare. What a nice find.” contains 10 words and 7 distinct words, giving a ratio of 7/10 = 0.7. section 11.3.1
A practical implementation follows three steps:
- Split the text into candidate words. section 11.5.3
- Clean surrounding punctuation and ignore empty words. section 11.5.3
- Count all valid words and store unique words in a Python set, then divide the unique count by the total count. section 11.5.3
Listing 11.4 Different words divided by total number of words
def different_to_total(text):
'''
text is a string of text.
Return the number of unique words in text
divided by the total number of words in text.
Do not count empty words as words.
Do not include surrounding punctuation.
>>> different_to_total('A pearl! Pearl! Lustrous pearl! \
Rare. What a nice find.')
0.7
'''
words = text.split() #1
total = 0 #2
unique = set() #3
for word in words:
word = clean_word(word) #4
if word != '':
total += 1
unique.add(word) #5
return len(unique) / total #6
Shows the complete Python function, including cleaning, duplicate removal, empty-word filtering, and the final division. section 11.5.3
