Term frequency (TF) starts with a count: how many times does a term occur in a document? To compare documents of different lengths, divide that count by the document’s total number of terms. The result is normalized term frequency—a fraction of the document’s tokens, rather than a raw count. section 3.1
>>> len(counts) #1 18 >>> counts.sum() 23 >>> len(tokens) #2 23 >>> counts / counts.sum() #3 , 0.217391 and 0.086957 It 0.043478 has 0.043478 also 0.043478 ...
The Python example sums token counts to get 23 terms, then divides each count by 23 to produce normalized frequencies. section 3.1
Compare documents of different lengths
In two tokenized documents about algorithmic bias, “bias” has a normalized TF of 0.0167 in a 484-token introduction and 0.0022 in a 9,550-token discussion of discrimination. This comparison uses each document’s own length as the denominator. By contrast, “and” scores about 0.03 in both documents—frequent, but not necessarily informative about either document’s subject. section 3.6
>>> intro_tf = {}
>>> disc_tf = {}
>>> intro_counts = Counter(intro_tokens)
>>> intro_tf['bias'] = intro_counts['bias'] / intro_total
>>> disc_counts = Counter(disc_tokens)
>>> disc_tf['bias'] = disc_counts['bias'] / disc_total
>>> 'Term Frequency of "bias" in intro is:{:.4f}'.format(intro_tf['bias'])
Term Frequency of "bias" in intro is:0.0167
>>> 'Term Frequency of "bias" in discrimination chapter is: {:.4f}'\
... .format(disc_tf['bias'])
'Term Frequency of "bias" in discrimination chapter is: 0.0022'
Inspect the `Counter` calculations for “bias”: count the term in each token list, divide by that list’s length, and compare the printed results. section 3.6
TF is useful in bag-of-words document vectors, but frequency within one document is not the same as importance across a collection. In a short sentence, “justice” occupies about 4% of the tokens, yet that share alone may understate its significance. TF–IDF adds inverse document frequency to account for how widely a term appears across documents; logarithmic scaling is also discussed for moderating frequency differences. Use TF when the within-document share is the question, and consider corpus-wide weighting when ranking terms for search or other NLP tasks. section 3.1 section 3.6 section 3.6.1
