How deeply does Outlier Detection in Python cover rare word?

Explains normalization, collection-wide counting, minimum frequency, and mean frequency, while cautioning against complex false-positive-prone methods.

Rare word

As explained in Outlier Detection in Python

A rare word is a word that occurs substantially less often than other words in a reference text collection.

A rare word occurs substantially less often than other words in a reference text collection. Detecting it requires comparing word frequencies across the full collection, not merely within one document. section 9.3.1

A practical detection workflow:

Normalize text by lowercasing words and removing nonalphabetic content so different cases do not create separate counts. section 9.3.1
Count each word across all records, then assign every document the frequency of its least-frequent word. section 9.3.1
Also consider the number of rare words or the document’s mean word frequency, because one unusual word may be less informative than a broader pattern. section 9.3.1

20 Newsgroups

Among 11,314 cleaned documents, 121 contained a word used only once. That threshold may be less useful than searching for documents containing many rare words. section 9.3.1

How Outlier Detection in Python covers it

liveBook manning.com

Read Outlier Detection in Python on liveBook Get the book at manning.com

Related concepts

frequencymeasured by · Frequency is the count or rate at which an item, event, outcome, or periodic cycle occurs within a specified collection or interval.
rare featureequivalent to · A rare feature is an input attribute value that occurs substantially less often than other values of the same feature.
rare categorycontrasts with · A rare category is a class or entity represented by substantially fewer observations than other categories in a dataset.

This explainer was generated by Manning's knowledge graph from the text of Outlier Detection in Python. The section links point to the parts of the book it draws on.