Token frequency counts how often a token occurs in a text collection; relative frequency expresses that count as a rate. These statistics help inspect corpora and choose an n-gram order. section 2.7 section 3.5
Counting words in the Brown Corpus
Download NLTK’s Brown Corpus, remove punctuation, lowercase the remaining words, and pass them to Python’s Counter. The ten most common tokens begin with “the,” “of,” and “and,” with counts of 69,971, 36,412, and 28,853. section 3.5
Frequency also reveals a broad distributional pattern:
- Ranks tokens by occurrence count in descending order. section 3.5
- Shows that, in a sufficiently large corpus, the top-ranked word occurs about twice as often as the second and four times as often as the fourth. section 3.5
>>> from collections import Counter
>>> puncs = set((',', '.', '--', '-', '!', '?',
... ':', ';', '``', "''", '(', ')', '[', ']'))
>>> word_list = (x.lower() for x in brown.words() if x not in puncs)
>>> token_counts = Counter(word_list)
>>> token_counts.most_common(10)
[('the', 69971),
('of', 36412),
('and', 28853),
('to', 26158),
('a', 23195),
('in', 21337),
('that', 10594),
('is', 10109),
('was', 9815),
('he', 9548)]
Inspect a runnable Counter example for lowercase token frequencies after punctuation filtering. section 3.5
