A token count vector turns a document into numbers a program can compare and calculate with. Each position belongs to one token in a shared vocabulary; its value is the number of times that token occurs in the document. A missing token gets zero. The vector records occurrences, not the order of words or their meaning. section 3.1 section 1.1.1
Count a sentence
In “It has also arisen in criminal justice, healthcare, and hiring, compounding existing racial, economic, and gender biases,” a tokenizer treats commas as tokens. The comma has count 5, while “and” has count 2. Try `tokens = [t.text for t in nlp(sentence)]`, then `counts = Counter(tokens)` and `counts.most_common(3)`. A pandas Series made from `counts` lets you inspect a named value with `counts['justice']` and divide counts by their total. section 3.1
For several documents, build the vocabulary across the whole corpus before assigning columns. Then count each document against those same columns, filling absent tokens with zeros. Lowercasing and sorting the vocabulary make the columns easier to keep consistent. In a 16-document example about algorithmic bias, the resulting table has 246 token columns; each row is one count vector. section 3.2
Listing 3.2 The built-in Python Counter class
>>> count_vectors = []
>>> for tokens in docs_tokens:
... count_vectors.append(Counter(tokens))
>>> tf = pd.DataFrame(count_vectors) #1
>>> tf = tf.T.sort_index().T
>>> tf = tf.fillna(0).astype(int)
>>> tf
" 's , ... within world wrongful
0 0 0 1 ... 0 0 0
1 0 0 3 ... 0 0 0
2 0 0 5 ... 0 0 0
3 2 0 0 ... 0 0 0
4 0 1 1 ... 0 0 0
5 0 0 0 ... 0 0 0
6 0 0 4 ... 0 1 0
...
11 0 0 1 ... 0 0 1
12 0 0 3 ... 0 0 0
13 0 0 1 ... 0 0 0
14 0 0 2 ... 0 0 0
15 2 0 4 ... 1 0 0
16 rows × 246 columns
The Python listing builds a document-by-token DataFrame from `Counter` objects, sorts its token columns, and fills missing counts with zero. section 3.2
Counts provide a rough gist, but the largest values need not identify the most useful words: commas and common words can dominate. Term frequency and TF–IDF offer ways to scale occurrence information using document statistics. An embedding is a different representation intended to capture semantics rather than only occurrence counts. section 3.1 section 3.2 section 1.1.1
The same vector can also be read as coordinates: with only “vector” and “space” in the vocabulary, counts of two and one place a document at (2, 1). More vocabulary terms simply require more dimensions. section 3.2.3
