A vocabulary is the set of unique tokens found in a corpus. Tokenization produces those tokens, which may represent words, subwords, or characters. section 7.8
A vocabulary supports text preparation in several ways:
- Collect unique tokens from the corpus after tokenization. section 7.8
- Optionally apply upper and lower frequency thresholds to select tokens. section 7.9
- Use the resulting vocabulary as the shared dimension for sentence vectors in a bag-of-words representation. section 7.9
- Assign zero to vocabulary terms absent from a particular sentence. section 7.9
Two-sentence vocabulary
For “It is raining heavily” and “We should eat fruits,” the vocabulary contains every unique word from both sentences. Each sentence becomes a vector containing its word frequencies, with zeros for words it does not contain. section 7.9
Figure 7.6 We are assuming that in the vocabulary only two sentences are present and the first sentence will be represented as shown.
Visualizes how a sentence is represented against a vocabulary containing the words from both example sentences. section 7.9
