liveBook · Manning

Tiktoken

As explained in Build a DeepSeek Model (From Scratch)

tiktoken is a fast tokenization library that implements byte-pair encoding for language-model text.

tiktoken is a fast Byte-Pair Encoding tokenizer for converting language-model text into integer token IDs. For English-only TinyStories data, the pretrained GPT-2 encoding provides a practical alternative to training a tokenizer or using DeepSeek's 100k-plus vocabulary. section 6.1.1

A typical preprocessing pipeline uses tiktoken to:

  • Load the standard gpt2 encoding and obtain its 50,257-token vocabulary size for model configuration. section 6.1.1 section 6.1.3
  • Encode each story into integer IDs, append the end-of-text token, and concatenate documents into one training stream. section 6.1.3
  • Store the resulting IDs as uint16 values, then write train and validation streams to memory-mapped binary files. section 6.1.3
Listing 6.2 Imports and configuration in prepare.py
import os
import json
from typing import List
import numpy as np
from datasets import load_dataset
import tiktoken
from tqdm import tqdm
 
# --- Configuration ---
DATASET_NAME = "roneneldan/TinyStories" #A
TOKENIZER_NAME = "gpt2" #B
OUTPUT_DIR = "data/tinystories_tokenized"
VAL_RATIO = 0.05 #C

The configuration listing shows how the dataset name, tokenizer name, output directory, and validation ratio become explicit preprocessing parameters. section 6.1.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Defines BPE tokenization and applies encoding, vocabulary extraction, separators, and storage, without detailing BPE merge training.
Examples
Uses TinyStories, the GPT-2 encoding, end-of-text separators, uint16 arrays, and train.bin and val.bin output files.
What it assumes
Assumes familiarity with datasets and NumPy while grounding tokenizer initialization, integer IDs, vocabulary size, and document separators.
What you can do after
Build a tokenized training corpus with vocabulary metadata, separated documents, validation data, and memory-efficient binary storage.
How it connects
Links tokenization to byte-pair encoding, vocabulary size, document boundaries, validation splits, NumPy arrays, and memory-mapped files.

Also explained in

Compare how 3 books explain tiktoken →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.