An end-of-text token is a special token appended after each document to preserve document boundaries when separate texts are concatenated into one training stream. It gives the model an explicit separator instead of silently merging unrelated sequences. section 6.1.3
Use the tokenization pipeline to:
- Read the tokenizer’s end-of-text ID through enc.eot_token. section 6.1.3
- Encode each document, append the ID, and extend a single flat token stream. section 6.1.3
- Store the resulting IDs as uint16 when the vocabulary fits within that range. section 6.1.3
Document stream construction
For documents A and B, encode A, append <|endoftext|>, then encode B and append the token again. The resulting stream retains both boundaries while remaining suitable for binary training data and memory-mapped storage. section 6.1.3
Listing 6.4 The token encoding function
def encode_corpus(texts: List[str], enc: tiktoken.Encoding) -> np.ndarray:
"""Encodes a list of texts into a single flat stream of token IDs."""
all_ids = []
eot_token = enc.eot_token #A
for text in tqdm(texts, desc="Encoding texts"):
ids = enc.encode(text)
all_ids.extend(ids)
all_ids.append(eot_token) #B
return np.array(all_ids, dtype=np.uint16) #C
# Inside main():
# 4. Tokenize the datasets
train_ids = encode_corpus([ex['text'] for ex in train_dataset], enc)
val_ids = encode_corpus([ex['text'] for ex in val_dataset], enc)Inspect the compact encode_corpus implementation that obtains enc.eot_token, appends it after every text, and returns a typed NumPy array. section 6.1.3






