A token is a unit of text a language model processes. It might be a whole word, part of a word, or punctuation. Think of tokens as the pieces into which a sentence is cut before the model works with it: “Alice was beginning” could become three word-sized tokens, while “unbelievable” could be split into “un,” “believ,” and “able.” The choice depends on the tokenizer. section 11.2 section 11.9
Tokenization makes those pieces and builds a vocabulary of them. One approach starts with individual characters and an end-of-word marker, counts neighboring token pairs, and repeatedly merges frequent pairs. The result can include both single characters and larger pieces such as “ing.” Keeping reusable pieces lets a vocabulary handle many words without requiring every word to be a separate token. section 11.9
def tokenize(text, vocabulary_size):
words = text.split()
current_tokens = []
for word in words:
current_tokens.extend(list(word) + [‘</w>‘]) #A
vocabulary = set(current_tokens)
while len(vocabulary) < vocabulary_size:
bigram_counts = collections.defaultdict(int) #B
for i in range(len(current_tokens) - 1):
pair = (current_tokens[i], current_tokens[i+1])
bigram_counts[pair] += 1
if not bigram_counts:
break
most_frequent_bigram = max(bigram_counts, key=bigram_counts.get)
count = bigram_counts[most_frequent_bigram] #C
if count <= 1:
break
new_token = "".join(most_frequent_bigram)
next_tokens = []
i = 0
while i < len(current_tokens):
is_match = (i < len(current_tokens) - 1 and
current_tokens[i] == most_frequent_bigram[0] and
current_tokens[i+1] == most_frequent_bigram[1])
if is_match: #D
next_tokens.append(new_token)
i += 2
else:
next_tokens.append(current_tokens[i])
i += 1
current_tokens = next_tokens
vocabulary.add(new_token) #E
final_vocabulary = sorted(list(vocabulary))
token_ids = {token: i for i, token in enumerate(final_vocabulary)} #F
return final_vocabulary, token_ids
Follow the Python tokenizer from characters and end-of-word markers through pair counts and merges to see how larger tokens are formed. section 11.9
Each vocabulary token gets a numerical ID. Replacing the tokens in a training text with their IDs produces a token stream the model can process. Because the stream is too long to process at once, it is divided into training examples that fit a context window—the maximum number of tokens the model can examine in one pass. section 11.10 section 11.4
From pieces to predictions
A tiny dataset of five sentences contains 29 word-sized tokens. Counting what follows “Alice” gives two possible next tokens: “was” with probability 0.67 and “and” with probability 0.33. This word-level example shows why token boundaries matter: predictions concern the next token, which need not always be a whole word. In a Transformer, self-attention also lets a token draw on other tokens in its current batch; when reading “Alice was beginning to get very tired,” “tired” can draw on “Alice” and “very.” section 11.2 section 11.17
Token count describes the amount of training text, not the number of distinct entries in its vocabulary. More training tokens provide more examples to learn from; the vocabulary determines which pieces receive IDs. When inspecting model input, check both the pieces produced by tokenization and how many fit in the context window. section 11.3 section 11.10
