Zipf's law describes a recurring pattern in ranked data: item frequency decreases approximately with rank. On a log-log plot, frequencies often form a near-straight line, making deviations worth investigating. section 3.5
The pattern appears in word counts and in broader systems such as city populations, economic output, resource distribution, and social networks. In language processing, it explains why a few words dominate counts while many occur rarely. section 3.5
Apply the observation to language data by:
- Plotting frequency against rank and investigating points that depart from the expected line. section 3.5
Zipf's law also motivates logarithmic scaling in TF–IDF. Applying logarithms to term frequency and inverse document frequency reduces the disproportionate effect of large count differences, producing more uniformly distributed scores. section 3.6.1
Figure 3.4 City population distribution27
Compare the ranked distribution of US city populations with the same kind of frequency pattern found in word usage. section 3.5
>>> log_tf = log(term_occurences_in_doc) -\ ... log(num_terms_in_doc) #1 >>> log_log_idf = log(log(total_num_docs) -\ ... log(num_docs_containing_term)) #2 >>> log_tf_idf = log_tf + log_log_idf #3
Inspect a compact implementation that applies logarithmic transformations to term frequency, document frequency, and their combined TF–IDF score. section 3.6.1
