Surprise measures how unexpected an observed token is. In document analysis, rare tokens carry more information because they distinguish one document from the rest of a corpus. section 3.6
Inverse document frequency (IDF) turns this intuition into a rarity measure: divide the total number of documents by the number containing the term. Common words receive little distinguishing weight; corpus-specific words receive more. section 3.6
Comparing words across documents
In two documents, “and” appears in both, so its IDF is 2/2 = 1. “bias” also appears in both and receives the same IDF. “black” appears in only one, so its IDF is 2/1 = 2. Multiplying term frequency by IDF gives TF-IDF, increasing the weight of words that are both present and distinctive. section 3.6
>>> DATA_DIR = ('https://gitlab.com/tangibleai/nlpia/'
... '-/raw/master/src/nlpia/data')
>>> url = DATA_DIR + '/bias_discrimination.txt'
>>> bias_discrimination = requests.get(url).content.decode()
>>> intro_tokens = [t.text for t in nlp(bias_intro.lower())]
>>> disc_tokens = [t.text for t in nlp(bias_discrimination.lower())]
>>> intro_total = len(intro_tokens)
>>> intro_total
484
>>> disc_total = len(disc_tokens)
>>> disc_total
9550
Inspect the document token counts used to compare term frequencies before applying surprise-based weighting. section 3.6
