A vocabulary maps the tokens a text system recognizes to integer IDs. After text is split into words or characters, those IDs give a model a numeric input. Building the mapping from training data keeps it focused on tokens that occur in the data of interest, rather than every possible character or word. section 14.2 section 14.2.1
What happens to a missing word?
Suppose a word vocabulary contains `"[UNK]": 0`, `"the": 1`, and `"quick": 2`. Looking up `the quick lazy` gives `[1, 2, 0]` if `lazy` is absent: `[UNK]` supplies an ID even for a token seen only at test time. Standardization matters too. Without lowercasing, `The` and `the` are different lookup keys. section 14.2 section 14.2.1
A practical way to build a vocabulary is to standardize the training text, split it into tokens, count them, reserve an `[UNK]` entry, and assign indices to the most frequent tokens up to a chosen limit. Use the same standardization and splitting rules when indexing new text; otherwise familiar-looking words may become unknown. section 14.2 section 14.2.1
Listing 14.4 Computing a word-level vocabulary
def compute_word_vocabulary(inputs, max_size): word_counts = collections.Counter() for x in inputs: x = x.lower() tokens = re.findall(r"[\w]+|[.,!?;]", x) word_counts.update(tokens) vocabulary = ["[UNK]"] most_common = word_counts.most_common(max_size - len(vocabulary)) for token, count in most_common: vocabulary.append(token) return dict((token, i) for i, token in enumerate(vocabulary))
The word-vocabulary function lowercases input, counts split tokens, reserves `[UNK]`, and indexes the most common tokens within `max_size`. section 14.2.1
Vocabulary size is a trade-off, not just a measure of coverage. On the same novel, a character tokenizer uses 64 entries but produces a 63-ID sample; a word tokenizer fills 2,000 entries yet produces only 13 IDs for that sample. Covering every word would require about 17,000 entries. Limiting the vocabulary also helps limit model parameters. section 14.2.1
The vocabulary determines the shape of several downstream representations:
- In a bag-of-words representation, each position records whether its vocabulary term appears. A 20,000-word setup produces 20,000-number vectors for review classification. section 14.4.1
- In a language model, vocabulary size determines how many next-token probabilities are predicted at each step. Predicting one token at a time avoids treating every possible output sequence as its own class. section 15.1
- In a character-level model, the same vocabulary size sets the embedding input range and softmax output size. One Shakespeare example learns a 67-character vocabulary with Keras `TextVectorization`. section 15.1.1
Rule of thumb: choose word tokens when shorter sequences matter, but watch vocabulary growth; choose character tokens when a small vocabulary matters, but expect longer sequences. Keep the tokenizer and its learned vocabulary together so later inputs receive the intended IDs. section 14.2.1 section 14.2
