A bag-of-words (BOW) model turns a sentence or document into a record of which words occur, or how often they occur. Imagine sorting tokens into labeled bins and counting each stack: the counts survive, but the original sequence does not. This gives text a numerical representation that can support document retrieval and other NLP tasks. section 1.5.3 section 3.1
To make documents comparable:
- Tokenize each document and establish a vocabulary. Keep the vocabulary’s order consistent across documents so each vector position refers to the same token. section 2.5.2 section 3.1
- Choose binary values for presence or absence, or counts for occurrences. Dividing counts by the total number of tokens gives word frequencies. section 3.1
- Inspect the tokens before interpreting their counts. Punctuation and common words can dominate an unfiltered bag. section 3.1
Count a sentence
Using spaCy’s `en_core_web_sm` model, tokenize “It has also arisen in criminal justice, healthcare, and hiring, compounding existing racial, economic, and gender biases”. Then `Counter(token.text for token in nlp(sentence))` gives five commas and two occurrences of `and`. The 23 tokens yield 18 distinct counted tokens; punctuation’s frequency is 5/23. These are properties of the tokens, not a judgment about which words matter. section 3.1
>>> len(counts) #1 18 >>> counts.sum() 23 >>> len(tokens) #2 23 >>> counts / counts.sum() #3 , 0.217391 and 0.086957 It 0.043478 has 0.043478 also 0.043478 ...
Inspect the count totals and the resulting frequencies, including punctuation, before using the vector downstream. section 3.1
Losing order is an acceptable approximation for some greetings and broad document topics, but it can erase the logic of a complex request. Rearrangements of “Good morning Rosa!” still suggest a greeting; rearranging a query that joins search conditions can change what it asks. For order-sensitive information, bags of n-grams retain short word sequences. TF–IDF takes a different next step: it weights word counts to better reflect their importance across documents. Neither turns a raw count into a reliable understanding of intent. section 1.7 section 3.0
