tiktoken is a fast Byte-Pair Encoding tokenizer for converting language-model text into integer token IDs. For English-only TinyStories data, the pretrained GPT-2 encoding provides a practical alternative to training a tokenizer or using DeepSeek's 100k-plus vocabulary. section 6.1.1
A typical preprocessing pipeline uses tiktoken to:
- Load the standard gpt2 encoding and obtain its 50,257-token vocabulary size for model configuration. section 6.1.1 section 6.1.3
- Encode each story into integer IDs, append the end-of-text token, and concatenate documents into one training stream. section 6.1.3
- Store the resulting IDs as uint16 values, then write train and validation streams to memory-mapped binary files. section 6.1.3
Listing 6.2 Imports and configuration in prepare.py
import os import json from typing import List import numpy as np from datasets import load_dataset import tiktoken from tqdm import tqdm # --- Configuration --- DATASET_NAME = "roneneldan/TinyStories" #A TOKENIZER_NAME = "gpt2" #B OUTPUT_DIR = "data/tinystories_tokenized" VAL_RATIO = 0.05 #C
The configuration listing shows how the dataset name, tokenizer name, output directory, and validation ratio become explicit preprocessing parameters. section 6.1.3


