A bag-of-words representation needs a count for each token in a sentence or document. Python’s `collections.Counter` does that bookkeeping: it is a dictionary whose keys are unique tokens and whose values are their occurrence counts. Those counts can become the nonzero entries of a document vector used in NLP search or machine learning. section 1.5.3 section 3.1
Count tokens in a sentence
Tokenize a sentence about algorithmic bias with spaCy, then count its tokens: ```python from collections import Counter tokens = [token.text for token in nlp(sentence)] bag_of_words = Counter(tokens) bag_of_words.most_common(3) # [(',', 5), ('and', 2), ('It', 1)] ``` Punctuation is counted because it appears in `tokens`. Inspecting the most common entries can reveal what dominates the count before the vector is used. section 3.1
Keep the next steps separate from counting:
- Check the tokenizer’s output. Splitting a greeting on spaces leaves attached punctuation and capitalization in the resulting tokens. section 1.5.3
- Give each vocabulary token a consistent vector position across documents. Counter’s displayed key order is not a substitute for that shared index. section 3.1
- Convert counts to a pandas Series when needed, then divide by their sum to obtain relative frequencies. In the example sentence, 23 tokens occupy 18 distinct count entries. section 3.1
>>> len(counts) #1 18 >>> counts.sum() 23 >>> len(tokens) #2 23 >>> counts / counts.sum() #3 , 0.217391 and 0.086957 It 0.043478 has 0.043478 also 0.043478 ...
The count totals and division show the transition from raw Counter entries to token-relative frequencies. section 3.1
Counter records frequency, not word sequence or meaning. A fixed vocabulary makes its counts comparable across documents; an out-of-vocabulary bucket can collect tokens omitted from that vocabulary, but cannot preserve their individual identities. For a further check, compare Counter-based document counts with `CountVectorizer.transform()` output and try converting both to matching DataFrames. section 3.1 section 3.9
