A vocabulary is the set of distinct tokens an NLP system recognizes. It is also called a lexicon, though lexicon can imply that additional information, such as spelling variations or parts of speech, is stored with each token. Tokens need not be ordinary words: punctuation, emojis, characters, and word pieces can all belong to a vocabulary. The tokenizer determines what counts as a token. section 2.1 section 3.2
For a collection of documents—a corpus—build a pipeline vocabulary by taking the union of the distinct tokens in every document. Choices made before that step matter. Lowercasing, for example, groups capitalized and lowercase forms under one token. Without such a choice, they occupy separate entries. section 3.2
Build a small vocabulary
In Python, try `docs = ["Swim and swim", "Swimming is fun"]`, then `tokens = [doc.lower().split() for doc in docs]` and `vocab = sorted(set(token for doc in tokens for token in doc))`. The result is `['and', 'fun', 'is', 'swim', 'swimming']`. Lowercasing combines “Swim” with “swim,” but does not combine “swim” with “swimming.” This whitespace-only example is a starting point; a tokenizer can also identify punctuation as tokens. section 3.2 section 2.3 section 2.1
A vocabulary gives document vectors consistent positions: one dimension per token, holding a count, weight, or other score. Its size is the number of distinct tokens, not the total number of token occurrences. In bag-of-words search, a vector can record whether each vocabulary token appears in a document. The token order is arbitrary, but it must stay consistent across documents. section 2.5.2 section 3.2.3
Listing 3.2 The built-in Python Counter class
>>> count_vectors = []
>>> for tokens in docs_tokens:
... count_vectors.append(Counter(tokens))
>>> tf = pd.DataFrame(count_vectors) #1
>>> tf = tf.T.sort_index().T
>>> tf = tf.fillna(0).astype(int)
>>> tf
" 's , ... within world wrongful
0 0 0 1 ... 0 0 0
1 0 0 3 ... 0 0 0
2 0 0 5 ... 0 0 0
3 2 0 0 ... 0 0 0
4 0 1 1 ... 0 0 0
5 0 0 0 ... 0 0 0
6 0 0 4 ... 0 1 0
...
11 0 0 1 ... 0 0 1
12 0 0 3 ... 0 0 0
13 0 0 1 ... 0 0 0
14 0 0 2 ... 0 0 0
15 2 0 4 ... 1 0 0
16 rows × 246 columns
Inspect the count table: each row represents a document, each column a vocabulary token, and each cell that token’s count. The 16 documents produce 246 columns. section 3.2
Vocabulary design is a trade-off. A token left out has no individual place in the vector; an out-of-vocabulary token can collect its occurrences, but loses distinctions between the words it replaces. Keeping every word form separate also misses useful relationships, while large vocabularies create large vectors. Subword tokenizers instead retain characters and combine frequent character sequences into tokens, allowing them to represent new words when those words contain no new characters. Choose tokenization and normalization with the task—and the information they discard—in mind. section 3.1 section 2.3 section 2.2.1 section 3.2.3
