A token is a small unit of text produced by splitting an input string. It might be a character, a word, punctuation, or a subword. Splitting is the first step toward turning text into the numbers a model can process. section 14.2
From a sentence to indices
Split “The quick brown fox jumped over the dog.” at word boundaries while keeping punctuation, and the result includes “The”, “quick”, and “.” as separate tokens. A vocabulary assigns each known token an integer index. With a vocabulary containing lowercase “the” but not uppercase “The”, a direct lookup maps “The” to 0; lowercasing before lookup avoids that mismatch. A word absent from the vocabulary still needs an unknown-token representation. section 14.2 section 14.2.1
vocabulary <- c("the", "quick", "brown", "fox", "jumped", "over", "dog", ".")
words <- split_words("The quick brown fox jumped over the lazy dog.")
indices <- match(words, vocabulary, nomatch = 0L)
indices
[1] 0 2 3 4 5 6 1 0 7 8
The lookup makes the numeric step visible: 0 marks tokens missing from this vocabulary, including uppercase “The” and “lazy”. section 14.2
The choice of token size changes what the model must handle:
- Characters need a small vocabulary but produce long sequences. In the Moby Dick example, 89 character terms cover the full input. section 14.2.1
- Words produce shorter sequences, but a limited vocabulary leaves uncommon words unmatched. The Moby Dick word example fills a 2,000-term vocabulary without covering every word. section 14.2.1
- Subwords offer another choice. Byte-pair encoding starts with characters and repeatedly merges frequent adjacent pairs; “o” and “w”, for example, can become one “ow” token. section 14.2.2
Tokens remain the units of work after preprocessing. A language model predicts a token from past tokens, and generated tokens can be fed back in to continue a sequence. In a Transformer, attention uses token embeddings to draw information selectively from elsewhere in the sequence. section 15.6 section 16.6
