A token is one unit in a sequence made from raw data. In text, it can be a character, word, subword, or punctuation mark. Neural networks need numerical input, so tokenization is followed by assigning tokens integer indexes and, when needed, turning those indexes into vector representations. section 8.2 section 12.1.2
The choice of token changes what the model sees:
- Word-level tokenization keeps words whole and is straightforward to implement, but punctuation must be separated so “way” and “way.” do not become different tokens. section 8.3 section 8.3.1
- Subword tokenization keeps frequent words whole and splits less common or complex words into pieces. It sits between character-level and word-level approaches. section 10.1
Turn a novel into token indexes
For Anna Karenina, replace line breaks with spaces, lowercase the text, space out punctuation, and split it. The result contains 437,098 tokens but 12,778 unique tokens. Sort unique tokens by frequency, then build both token-to-index and index-to-token dictionaries: comma maps to 0, period to 1, and “the” to 2. The full text can now become an integer sequence. section 8.3.1
clean_text=text.lower().replace("\n", " ") #1
clean_text=clean_text.replace("-", " ") #2
for x in ",.:;?!$()/_&%*@'`":
clean_text=clean_text.replace(f"{x}", f" {x} ")
clean_text=clean_text.replace('"', ' " ') #3
text=clean_text.split()
The cleaning code shows how lowercasing, replacing line breaks, and spacing punctuation determine the resulting word-level tokens. section 8.3.1
Those indexes become model inputs. A text-generation Transformer groups them into sequences of up to 128 indexes; an English–French Transformer instead starts with subword indexes, then uses word embeddings and positional encodings to represent both tokens and their order. section 12.1.2 section 10.0
Tokens beyond text
A music Transformer turns MIDI notes into event tokens for note-on, note-off, time-shift, and velocity. Two time-shift tokens with values 99 and 2 represent 1 second followed by 0.03 seconds. After indexing its 388 possible events, the model trains on sequences that predict the next music token. section 14.1.1 section 14.1.3
