A token is one unit in a text sequence. Before text can become numeric model input, it must be split into units such as characters or words. The choice changes the sequence the model receives. section 14.2
def split_words(text): return re.findall(r"[\w]+|[.,!?;]", text)
This regex-based splitter returns words and selected punctuation marks as separate tokens. section 14.2
Splitting produces strings, not yet numeric inputs. A vocabulary maps each token to an integer index. An unknown-token entry can supply an index when a string is absent from the vocabulary. section 14.2
vocabulary = { "[UNK]": 0, "the": 1, "quick": 2, "brown": 3, "fox": 4, "jumped": 5, "over": 6, "dog": 7, ".": 8, } words = split_words("The quick brown fox jumped over the lazy dog.") indices = [vocabulary.get(word, 0) for word in words]
The lookup maps known words to indices and falls back to [UNK] for missing entries. Here, both capitalized “The” and “lazy” take that fallback. section 14.2
A token in context
In “The train left the station on time,” the meaning of “station” depends on nearby words. Self-attention lets its representation draw on “train” and other tokens in the sequence, including itself. section 15.3.2
Tokens need not be whole words: pretrained Transformer models also embed subwords. A useful distinction is that tokenization decides the sequence units, while embeddings and self-attention determine how those units are represented and related. section 14.2 section 15.5
