A bag-of-words vector counts words but discards their order. An n-gram keeps neighboring elements together as one feature: “ice cream” is a word 2-gram, while “was not” keeps a negation beside the word it changes. Elements can also be characters, syllables, or symbols; the choice depends on the task. section 2.3.1
Choose the elements and lengths before counting:
- Word n-grams retain short phrases for bag-of-words search. They need not be special expressions; recurring neighbors can be useful features. section 2.3.1
- Character n-grams capture fragments within words, making them useful for tasks involving unfamiliar names, language detection, or authorship attribution. section 3.4.1 section 5.2.4
- A wider n-gram range creates more features. On large corpora, that growth can make bag-of-n-grams vectors costly to work with. section 3.4
Keep a search phrase together
For the query “What is algorithmic bias?”, counting only individual words cannot represent “algorithmic bias” as a single feature. CountVectorizer(ngram_range=(1, 2)) counts both single words and adjacent word pairs. In a 16-document example, its vectors have 616 columns; the “algorithmic bias” column identifies sentences containing that pair. The extra columns are the cost of preserving this local context. section 3.4
>>> ngram_vectorizer = CountVectorizer(ngram_range=(1, 2))
>>> ngram_vectors = ngram_vectorizer.fit_transform(corpus)
>>> ngram_vectors
<16x616 sparse matrix of type '<class 'numpy.int64'>'
with 772 stored elements in Compressed Sparse Row format>
Inspect the CountVectorizer setting and resulting sparse-matrix shape when both 1-grams and 2-grams are counted. section 3.4
For names, switch the analyzer to characters instead of splitting into words. A TfidfVectorizer configured with analyzer='char' and ngram_range=(1, 3) represents names using one-, two-, and three-character fragments. Such features let a model work with names absent from its training examples. In the shown configuration, use_idf=False, so it does not apply inverse-document-frequency weighting. section 5.2.4
>>> from sklearn.feature_extraction.text import TfidfVectorizer >>> vectorizer = TfidfVectorizer( ... use_idf=False, #1 ... analyzer='char', ... ngram_range=(1, 3) #2 ... ) >>> vectorizer TfidfVectorizer(analyzer='char', ngram_range=(1, 3), use_idf=False)
See the character analyzer, n-gram range, and disabled IDF weighting configured together. section 5.2.4
