Tokenization splits text into units called tokens: characters, words, punctuation marks, or parts of words. A neural network cannot use raw text directly, so tokens are assigned integer indexes before they become model inputs. The choice of token size affects vocabulary size, training cost, and how much meaning an individual token carries. section 8.2 section 8.2.1
Three common choices make different tradeoffs:
- Character tokenization splits every character, including spaces and punctuation. It keeps the set of unique tokens small, but individual characters often carry little meaning. section 8.2.1
- Word tokenization separates words and punctuation. Words carry meaning, but a larger vocabulary can increase model parameters and training time. section 8.2.1
- Subword tokenization keeps frequent words whole while splitting less common or complex words into pieces. It offers a balance between character and word tokenization. section 10.1
From phrase to model input
With word tokenization, “It is unbelievably good!” becomes ['It', 'is', 'unbelievably', 'good', '!']. For next-token training, assign each distinct token an integer, cut the resulting index stream into equal-length inputs, and shift each input one token right to make its target. An LSTM example uses inputs of 100 indexes; a GPT example uses 128. section 8.2.1 section 8.1.3 section 12.0
For a simple tokenizer built from scratch, lowercase the text, replace line breaks with spaces, put spaces around punctuation, and split on whitespace. In a three-novel example, this yields 10,599 distinct tokens before adding an unknown-token entry. Count the tokens, build token-to-index and index-to-token dictionaries, then convert the training text to indexes. Cleaning matters: without added spaces, punctuation stays attached to words. section 12.2.1
Listing 12.3 Adding spaces around punctuation marks
text=text.lower().replace("\n", " ") #1
chars=set(text.lower())
punctuations=[i for i in chars if i.isalpha()==False
and i.isdigit()==False] #2
print(punctuations)
for x in punctuations:
text=text.replace(f"{x}", f" {x} ") #3
text_tokenized=text.split()
unique_tokens=set(text_tokenized)
print(len(unique_tokens)) #4
The listing shows the small but important preprocessing step that separates punctuation before whitespace splitting. section 12.2.1
Subwords for translation
A pretrained XLM tokenizer splits “I don't speak French.” into pieces including 'fr' and 'ench'; the paired French phrase splits “français” into 'franc' and 'ais'. Build separate English and French index dictionaries, reserve BOS and EOS for sentence boundaries and PAD and UNK for padding and unknown tokens, and keep reverse dictionaries for converting indexes back to tokens. section 10.1.1
The same preparation pattern extends beyond text. A music Transformer turns MIDI notes into note-on, note-off, time-shift, and velocity event tokens, then indexes the events. Time increments and velocity bins keep the event vocabulary manageable. The detailed MIDI-to-event conversion is handled by a processor module rather than developed from scratch. section 14.1.1 section 14.2.2
