Word frequency counts how often a word occurs across a text collection. For document-level analysis, frequencies can reveal records containing unusually rare vocabulary. section 9.3.1
A practical frequency workflow:
- Normalize text by lowercasing words and removing nonalphabetic tokens so equivalent forms match. section 9.3.1
- Count words across the full collection, then calculate each document’s least-word frequency or mean word frequency. section 9.3.1
- Interpret rare-word signals cautiously because complex text features can flag many false positives. section 9.3.1
Rare vocabulary in newsgroup messages
Among 11,314 messages, 121 contained words used only once. Measuring the mean frequency of words in each message can instead identify documents with an unusually high proportion of rare words. section 9.3.1
Listing 9.9 Finding the frequency of the rarest word in each record
import pandas as pd
from sklearn.datasets import fetch_20newsgroups
from collections import Counter
import statistics
def clean_data(x): #1
return [c.lower() for c in x.split() if c.isalpha()]
data = fetch_20newsgroups(
remove=('headers', 'footers', 'quotes')) #2
df = pd.DataFrame({"Content": data['data']})
df['Cleaned Content'] = df['Content'].apply(clean_data)
full_text = [] #3
for v in df.index:
full_text.extend(df.loc[v, 'Cleaned Content'])
c = Counter(full_text) #4
min_frequency_arr = [] #5
mean_frequency_arr = []
for v in df['Cleaned Content']:
if len(v) == 0: #6
min_frequency_arr.append(0)
mean_frequency_arr.append(0)
else: #7
min_frequency_arr.append(min([c[word] for word in v]))
mean_frequency_arr.append(statistics.mean([c[word] for word in v]))
df['Min Frequency Word'] = min_frequency_arr
df['Mean Frequency Word'] = mean_frequency_arr
Shows Python code for cleaning records, building corpus-wide counts with Counter, and deriving rarest-word and mean-frequency features. section 9.3.1
