Tokenization breaks text into smaller units called tokens so a language model can process it. Tokens may be words, character pieces, or punctuation rather than complete words. section 11.9
A practical tokenization workflow moves through these steps:
- Split training text into initial pieces, such as individual characters, and append an end-of-word marker like </w> so words remain distinguishable. section 11.9
- Count adjacent token pairs, called bigrams, then repeatedly merge the most frequent pair to form larger reusable tokens. section 11.9
- Assign each vocabulary token an integer ID, turning tokenized text into a numerical token stream for model training. section 11.10
- Divide the stream into batches that fit the model's context window before training or inference. section 11.10
From words to a token stream
The sentence “Alice was beginning to get very tired of sitting” can initially be represented as word tokens. A word such as unbelievable may instead become un, believ, able, allowing related pieces to recur across words. With a character-based procedure, frequent pairs are merged step by step; the resulting vocabulary can include pieces such as in, ing, ers, th, and punctuation. Those tokens receive IDs and become a numerical sequence, which is then sliced into training batches. section 11.9 section 11.10
Figure 11.19 Tokenized training data as a token stream
Visualizes tokenized training data as a stream of token IDs. section 11.10
def batch_tokens(token_stream, context_window_size):
batches = []
start_index = 0
while True:
end_index = start_index + context_window_size #A
if end_index > len(token_stream): #B
break
batch = token_stream[start_index:end_index] #C
batches.append(batch)
start_index += 1 #D
return batches
Shows Python-style sliding-window batching that extracts sequences from a continuous token stream. section 11.10
