liveBook · Manning

Tiktoken

tiktoken is an OpenAI-developed library that efficiently encodes text using model-specific byte-pair-encoding tokenizers and supports token counting.

How 3 Manning books explain it

Essential GraphRAG

tiktoken is a tokenization library developed by OpenAI for encoding text and counting language-model tokens.

How it is used
Uses token counts to detect oversized sections before splitting documents for retrieval and LLM generation.
How deep it goes
Defines and demonstrates model-specific encoding and counting, then stops short of explaining tokenizer internals.
Examples
Counts tokens across extracted sections, including sections with 154, 4,186, and 2,703 tokens.
What you can do after
Apply it via tiktoken to measure text length and decide when retrieval sections require further splitting.

Read how Essential GraphRAG explains tiktoken →

Build a DeepSeek Model (From Scratch)

tiktoken is a fast tokenization library that implements byte-pair encoding for language-model text.

How it is used
Prepares text corpora for language-model training by converting documents into compact, numerically configured token streams.
How deep it goes
Defines BPE tokenization and applies encoding, vocabulary extraction, separators, and storage, without detailing BPE merge training.
Examples
Uses TinyStories, the GPT-2 encoding, end-of-text separators, uint16 arrays, and train.bin and val.bin output files.
What you can do after
Build a tokenized training corpus with vocabulary metadata, separated documents, validation data, and memory-efficient binary storage.

Read how Build a DeepSeek Model (From Scratch) explains tiktoken →

Generative AI in Action

tiktoken is an OpenAI library for encoding text into the token sequences used by OpenAI language models.

How it is used
Supports prompt-cost and context planning by counting the tokens processed or generated by language models.
How deep it goes
Explains BPE, direct and model-derived encoding selection, counting, and reversible decoding without descending into algorithmic internals.
Examples
Uses Python helpers, the prompt “I have a white dog named Champ,” token ID lists, and OpenAI encoding tables.
What you can do after
Apply it via Python to count, encode, and decode text using an encoding selected for an OpenAI model.

Read how Generative AI in Action explains tiktoken →

Also covered in

What to know first

Related concepts

Questions about tiktoken

What is tiktoken?

tiktoken is an OpenAI-developed library that efficiently encodes text using model-specific byte-pair-encoding tokenizers and supports token counting.

Which Manning books explain tiktoken?

Essential GraphRAG, Build a DeepSeek Model (From Scratch) and Generative AI in Action.

What should I know before learning tiktoken?

Manning books that teach tiktoken usually introduce context window first.

What kinds of tiktoken are there?

Kinds of tiktoken include js-tiktoken.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.