Indexing turns tokens into integer IDs so text can become numeric model input. A vocabulary maps each token to an ID; the mapping is reversible and can be used with different modeling approaches. section 14.2
vocabulary = { "[UNK]": 0, "the": 1, "quick": 2, "brown": 3, "fox": 4, "jumped": 5, "over": 6, "dog": 7, ".": 8, } words = split_words("The quick brown fox jumped over the lazy dog.") indices = [vocabulary.get(word, 0) for word in words]
This Python example looks up each word in a vocabulary and assigns ID 0 to tokens not found there. Here, both “The” and “lazy” map to the [UNK] ID. section 14.2
The resulting IDs can later be mapped into a learned latent space, such as through embeddings. Indexing itself does not perform that mapping. section 14.2 section 14.6
