Tokenization
Tokenization is the process of dividing text into discrete units called tokens for computational processing.
Learn Generative AI with PyTorch
The process of splitting text into discrete units such as characters, words, or subwords for modeling.
- How it is used
- Prepares text and music as indexed sequences for generation and translation models.
- How deep it goes
- Walks through word splitting and indexing, uses a pretrained subword tokenizer, and defers detailed MIDI event conversion.
- Examples
- Uses novel text, English–French phrase pairs, and MIDI note events to demonstrate tokenization.
- What you can do after
- Build word-level tokenization and indexing pipelines and apply pretrained XLM subword tokenization to English and French phrases.
Read how Learn Generative AI with PyTorch explains tokenization →
LLMs in Production
Tokenization is the process of dividing text into discrete units that can be represented and processed by a language model.
- How it is used
- Prepares text for language-model training and inference, with token choices determining what information reaches the model.
- How deep it goes
- Explains token units, unknown-token failures, task-specific trade-offs, and tokenizer training and configuration with code.
- Examples
- Uses code completion, whitespace-split word counts, arithmetic digit grouping, and a subword tokenizer training listing.
- What you can do after
- Apply tokenization via Hugging Face tools by training a subword tokenizer or configuring special tokens for batching.
How Large Language Models Work
Tokenization is the process of converting an input into discrete units or a sequence of numeric representations for model processing.
- How it is used
- Prepares text as numeric model input and shapes which content an LLM can represent efficiently.
- How deep it goes
- Explains normalization, segmentation, numeric mapping, vocabulary trade-offs, BPE, and capability risks without dense implementation detail.
- Examples
- Uses whitespace splitting, BPE pair merging, running versus runnin, digit-level arithmetic, multilingual token costs, and code examples.
- What you can do after
- Evaluate tokenizer effects on vocabulary, cost, language coverage, arithmetic, code, and application-specific representation.
Read how How Large Language Models Work explains tokenization →
Deep Learning with R, Third Edition
Word tokenization is the division of text into tokens corresponding primarily to words.
- How it is used
- Prepares text as integer data for classification and character-level language modeling.
- How deep it goes
- Builds character, word, and byte-pair tokenizers and compares vocabulary size, sequence length, and unknown-token handling.
- Examples
- Uses Moby Dick vocabulary comparisons, IMDb sentiment classification, and Shakespeare character prediction.
- What you can do after
- Build it from scratch in R or apply it via Keras TextVectorization to classification and language-model datasets.
Read how Deep Learning with R, Third Edition explains tokenization →
Software Testing with Generative AI
Tokenization is the process of converting text into discrete tokens that a language model can process as numeric identifiers.
- How it is used
- Supports prompt design and fine-tuning data preparation by translating text into the representation language models process.
- How deep it goes
- Explains token splitting, integer mapping, tokenizer alignment, and context-length truncation, but does not detail tokenizer algorithms.
- Examples
- Uses a city-list prompt, integer token sequences, and fine-tuning examples involving test-code instructions and outputs.
- What you can do after
- Evaluate prompt and training-data limits by estimating token boundaries, checking tokenizer alignment, and identifying context-length truncation risks.
Read how Software Testing with Generative AI explains tokenization →
Deep Learning with Python, Third Edition
The process of converting text into a sequence of discrete units such as words, characters, or punctuation marks.
- How it is used
- Prepares text as integer input for classification and pretrained models.
- How deep it goes
- Builds character and word tokenizers through normalization, splitting, vocabulary construction, and indexing; identifies subword tokenization without detailing its implementation.
- Examples
- Compares character and word vocabularies on Moby Dick and preprocesses IMDb reviews for classification.
- What you can do after
- Build a character or word tokenizer from scratch, including vocabulary lookup and unknown-token handling.
Read how Deep Learning with Python, Third Edition explains tokenization →
Timeless Algorithms, The Foundational Ideas
The process of dividing text into discrete units called tokens.
- How it is used
- Introduces tokenization as the step that converts text into identifiers before transformer processing.
- How deep it goes
- Defines token types and identifiers, then explains their connection to learned embeddings without detailing a tokenization algorithm.
- Examples
- Uses an illustrative pronoun-resolution calculation involving tokens for “trophy,” “box,” and “it.”
- What you can do after
- Evaluate and understand how token identifiers differ from learned vector representations.
Read how Timeless Algorithms, The Foundational Ideas explains tokenization →
Data Without Labels
Tokenization is the process of splitting text into discrete units such as words or punctuation marks called tokens.
- How it is used
- Prepares text as tokens for vocabularies and downstream text representations.
- How deep it goes
- Defines word, subword, and character tokens, surveys boundary methods, and demonstrates NLTK calls without detailing tokenizer internals.
- Examples
- Uses a sentence split into tokens, Python cleaning snippets, and tokenized customer reviews in a sentiment workflow.
- What you can do after
- Apply it via NLTK to inspect tokens and prepare cleaned text for further processing.
Build AI Drug Discovery Pipelines
Tokenization is the conversion of structured input into a sequence of discrete tokens for computational processing.
- How it is used
- Supports molecular autoencoders that compress and reconstruct SMILES strings for generative drug design.
- How deep it goes
- Explains vocabulary construction, substitutions, special tokens, encoding, batching, and the shift from characters to learned molecular subunits.
- Examples
- Uses CCO, benzene notation, chlorine, bromine, aromatic nitrogen, and explicit hydrogen as tokenization examples.
- What you can do after
- Build it from scratch by creating a Python/PyTorch vocabulary and encoding, decoding, and batching SMILES sequences.
Read how Build AI Drug Discovery Pipelines explains tokenization →
Hugging Face in Action
A tokenization strategy is a rule or method for dividing text into model-processing units.
- How it is used
- Prepares text and datasets as numerical inputs for NLP models, inference, visualization, and fine-tuning.
- How deep it goes
- Explains three tokenization strategies and dataset fields, then demonstrates batching, padding, truncation, and token-ID inspection.
- Examples
- Uses BERT subwords, word-level sentences, multilingual character examples, sentiment inputs, and mapped training datasets.
- What you can do after
- Apply it via Hugging Face AutoTokenizer to tokenize text, inspect tokens and IDs, and prepare padded datasets for model use.
Fight Fraud with Machine Learning
Text tokenization is the conversion of text into discrete units such as words, characters, or subwords for computational processing.
- How it is used
- Converts words into numeric vectors for transformer encoder input in language translation.
- How deep it goes
- Defines word-level one-hot tokenization and works through vocabulary and embedding-matrix dimensions, without covering subword methods.
- Examples
- Uses the corpus “let us fight fraud” to construct four one-hot word vectors and a 4×4 matrix.
- What you can do after
- Evaluate/understand how a sentence becomes numeric encoder input before transformer processing.
Read how Fight Fraud with Machine Learning explains tokenization →
Time Series Forecasting Using Foundation Models
Tokenization is the process of converting input data into discrete units suitable for sequence-model processing.
- How it is used
- Converts time-series values, lags, and calendar information into inputs that decoder or language models can process for forecasting.
- How deep it goes
- Explains feature construction, mean scaling, quantization, reversal, and precision trade-offs, while deferring implementation code and formal algorithm details.
- Examples
- Uses hourly calendar features, store-sales and road-traffic scales, mean-scaled series, and comparisons between 100-bin and five-bin vocabularies.
- What you can do after
- Apply it via Lag-Llama or Chronos by constructing features or scaled bins, then restoring Chronos outputs through dequantization and unscaling.
Read how Time Series Forecasting Using Foundation Models explains tokenization →
Natural Language Processing in Action, Second Edition
Tokenization is the process of dividing a character sequence into discrete lexical units such as words or punctuation.
- How it is used
- Turns unstructured text into numerical inputs for NLP pipelines, search, classification, information retrieval, and conversational systems.
- How deep it goes
- Traces segmentation from documents to tokens, compares word and subword units, and reaches vectorization and preprocessing tradeoffs.
- Examples
- Uses greetings such as Hi and Hello, confusable strings such as Hiking and Hell, emojis, logograms, and subword encodings.
- What you can do after
- Apply it via a configured tokenizer to represent text, detect keywords, support search, and prepare tokens for numerical encoding.
Read how Natural Language Processing in Action, Second Edition explains tokenization →
Build a Text-to-Image Generator (from Scratch)
Tokenization is the process of splitting text into discrete tokens such as words, punctuation marks, or subwords.
- How it is used
- Converts raw language into discrete units and numerical inputs for translation, transformers, and text-to-image generation.
- How deep it goes
- Explains splitting, vocabulary construction, special indices, padding, and embedding handoff, but defers tokenizer internals.
- Examples
- Uses German-English translation phrases, “Wie geht es dir,” animal prompts, and “panda with top hat reading a book.”
- What you can do after
- Apply it via spaCy or Transformers to tokenize prompts, manage vocabulary indices, and prepare model-ready embeddings.
Read how Build a Text-to-Image Generator (from Scratch) explains tokenization →
Machines that Think
A procedure for dividing text into discrete units for computational processing.
- How it is used
- Establishes the discrete units a language model receives and the distinctions its input representation can preserve.
- How deep it goes
- Explains segmentation trade-offs and contrasts learned tokenizer approaches without providing implementation instructions.
- Examples
- Uses a sentence split into words and punctuation and divides “playfulness” into reusable subwords.
- What you can do after
- Evaluate tokenization choices by weighing linguistic coverage, vocabulary size, sequence length, and preserved distinctions.
MongoDB 8.0 in Action, Third Edition
Tokenization is the process of dividing a stream of text into smaller units called tokens.
- How it is used
- Prepares document text and queries as searchable terms for full-text search.
- How deep it goes
- Explains splitting, normalization, filters, and analyzer selection, but leaves custom-analyzer construction to external documentation.
- Examples
- Uses two inspection documents, their shared indexed terms, a matching query, and a table of built-in analyzers.
- What you can do after
- Operate/configure it in Atlas Search by selecting indexing and query analyzers or defining a custom analyzer.
Read how MongoDB 8.0 in Action, Third Edition explains tokenization →
Build a Large Language Model (From Scratch)
The process of splitting raw text into tokens for further processing by a language model.
- How it is used
- Prepares text for LLM pretraining by producing tokens that can become vector inputs.
- How deep it goes
- Explains regular-expression tokenization, whitespace choices, and downstream encoding, while deferring detailed byte pair encoding implementation.
- Examples
- Demonstrates Python regular-expression splits, punctuation cleanup, whitespace filtering, and preprocessing a short story.
- What you can do after
- Build it from scratch with Python regular expressions and produce cleaned word-and-punctuation token lists for later encoding.
Read how Build a Large Language Model (From Scratch) explains tokenization →
AI-Powered Search
Tokenization is the process of segmenting text into units such as words, subwords, or punctuation symbols.
- How it is used
- Supports query interpretation, spelling correction, semantic encoding, and question-answering model preparation.
- How deep it goes
- Explains filtering, whole-query alternatives, model vocabulary, sliding windows, overlap, and padding without detailing tokenizer internals.
- Examples
- Uses filtered query logs, product-name misspellings, and question-answering contexts divided into overlapping fixed-size token windows.
- What you can do after
- Apply tokenization with NLTK or a Transformers tokenizer, while choosing between term-level and whole-query processing.
Rearchitecting LLMs
Tokenization is the process of converting text into discrete numerical units that a language model can process.
- How it is used
- Prepares raw text for knowledge-distillation training and checks prompt pairs for position-aligned comparisons.
- How deep it goes
- Defines token IDs and tensor shapes, then implements padding, truncation, attention masks, and labels without detailing tokenization algorithms.
- Examples
- Uses SlimPajama samples with 128-token padded or truncated inputs and paired prompts checked for equal token counts.
- What you can do after
- Apply it via a Hugging Face tokenizer to create input IDs, attention masks, and labels for training.
Elasticsearch in Action, Second Edition
Tokenization is the process of splitting a character stream into discrete tokens according to rules such as whitespace, delimiters, or character patterns.
- How it is used
- Prepares analyzed text fields for query matching by converting character streams into searchable tokens.
- How deep it goes
- Defines rule-based splitting and variants, but defers synonym handling, stemming, and other transformations to normalization.
- Examples
- Uses synopsis text, whitespace-separated words, movie-review punctuation, hyphenated terms, and alphanumeric strings as examples.
- What you can do after
- Evaluate tokenizer choices for searchable text and distinguish splitting from subsequent normalization.
Read how Elasticsearch in Action, Second Edition explains tokenization →
Build AI-Enhanced Web Apps
Tokenization is the conversion of text or other input into a sequence of tokens or numerical token identifiers.
- How it is used
- Supports efficient communication between prompts and LLMs while helping developers estimate interaction costs.
- How deep it goes
- Explains the input-output pipeline, numerical IDs, billing, and JavaScript token counting, but stops short of tokenizer internals.
- Examples
- Uses a translation prompt, a system message, token-counting code, and an end-to-end processing diagram.
- What you can do after
- Apply it via @dqbd/tiktoken to count model-specific prompt and system-message tokens.
Generative AI in Action
Tokenization is the process of converting text into a sequence of discrete tokens for language-model processing.
- How it is used
- Connects LLM processing and enterprise search design to token boundaries, context, cost, latency, and field behavior.
- How deep it goes
- Defines tokenization, explains iterative next-token generation, and stops short of algorithm internals.
- Examples
- Uses an eight-token dog sentence and contrasts Redis TagField exact matching with TextField tokenized searching.
- What you can do after
- Evaluate token costs and configure Redis fields for exact-match or tokenized search behavior.
Build an Advanced RAG Application (From Scratch)
Tokenization is the process of splitting text into words, phrases, or other meaningful units for language processing.
- How it is used
- Supports keyword retrieval through indexed terms and semantic search through sentence representations.
- How deep it goes
- Explains word, subword, and sentence units, then shows token iteration and inverted-index population without detailing tokenizer internals.
- Examples
- Uses machine-learning documents, a fox sentence, and sample sentences encoded with SentenceTransformer.
- What you can do after
- Apply it via an inverted index or Sentence Transformers to prepare text for keyword lookup or semantic retrieval.
Read how Build an Advanced RAG Application (From Scratch) explains tokenization →
Starting Data Analytics with Generative AI and Python
An algorithm that divides text into the token units consumed by a language model.
- How it is used
- Frames tokenization as the practical basis for estimating prompt capacity and managing generative AI context limits.
- How deep it goes
- Defines token units and gives an average ratio, but stops short of explaining algorithm internals or comparing specific methods.
- Examples
- Uses words, word parts, and punctuation marks as illustrative token units.
- What you can do after
- Evaluate when inputs may exceed context limits and recognize truncation through inconsistent responses or forgotten exchanges.
Read how Starting Data Analytics with Generative AI and Python explains tokenization →
AI Agents in Action, Second Edition
Tokenization is the process of converting text into discrete tokens that a language model can process.
- How it is used
- Supports LLM input preparation, cost estimation, context management, and quality control.
- How deep it goes
- Explains token pieces, vocabulary indices, generation, JSON overhead, and measurement, but stops short of tokenizer algorithm internals.
- Examples
- Compares regular text with JSON and recommends measuring counts using tiktoken or Agents SDK telemetry.
- What you can do after
- Apply it via tiktoken or Agents SDK telemetry to measure tokens and manage budgets, context windows, and prompt overhead.
Read how AI Agents in Action, Second Edition explains tokenization →
Grokking AI Algorithms, Second Edition
Tokenization is the process of breaking text into a sequence of discrete units for computational language processing.
- How it is used
- Prepares language-model text by converting it into vocabulary units, token IDs, streams, and context-window-sized batches.
- How deep it goes
- Explains character initialization, end-of-word markers, bigram counts, repeated BPE merges, vocabulary IDs, and batching.
- Examples
- Uses Alice's sentence, the word unbelievable, character markers, bigram tables, token-ID listings, token streams, and Python batching code.
- What you can do after
- Build it from scratch by merging frequent character pairs, assigning token IDs, and slicing token streams into training batches.
Read how Grokking AI Algorithms, Second Edition explains tokenization →
Applied Reinforcement Learning
Tokenization is the process of dividing text into discrete units called tokens.
- How it is used
- Converts input text into discrete units that a neural network can process.
- How deep it goes
- Defines token types, ID mapping, and an example split, but does not detail tokenizer internals.
- Examples
- Uses “Take aspirin today” to illustrate a possible split into tokens.
- What you can do after
- Understand how input text becomes token IDs before model processing.
Read how Applied Reinforcement Learning explains tokenization →
Write Powerful Rust Macros
Lexing is the process of converting raw source text into a sequence of lexical tokens.
- How it is used
- Explains the representation procedural macros consume and produce while positioning lexing before parsing.
- How deep it goes
- Defines tokenization and contrasts it with parsing, but stops short of detailed compiler internals.
- Examples
- Uses `1 + 11` and its TokenStream representation to demonstrate literals and punctuation.
- What you can do after
- Evaluate/understand how Rust represents macro input before parsing and code generation.
Local Image Generation
Tokenization is the process of breaking text into smaller units that can be mapped to numeric identifiers for model processing.
- How it is used
- Converts Stable Diffusion prompts into numeric sequences for the CLIP encoder and subsequent neural-network processing.
- How deep it goes
- Compares character and word representations, including their meaning and vocabulary trade-offs.
- Examples
- Uses a Jabberwocky sentence, ASCII character values, word vocabulary mapping, and learned text splitting.
- What you can do after
- Evaluate how character-based and word-based splits affect meaning and vocabulary size before model processing.
Coding with AI
Tokenization is the process of breaking text into smaller units such as words, punctuation marks, or phrases.
- How it is used
- Supports NLP tasks by converting text into units that can be counted, compared, classified, and analyzed.
- How deep it goes
- Defines word-based tokenization and demonstrates preprocessing and counting, while deferring deeper NLP treatment.
- Examples
- Uses a sentence-token list and a truck-frequency example to demonstrate punctuation handling, normalization, and word counting.
- What you can do after
- Apply it in a Python workflow that cleans text, counts word frequencies, and displays the most frequent words.
Just Use Postgres!
Tokenization is the process of splitting text into discrete tokens for linguistic or computational processing.
- How it is used
- Prepares document text for PostgreSQL full-text search by producing searchable tokens and lexemes.
- How deep it goes
- Explains parsing, token types, dictionaries, stemming, and stop-word removal, then stops short of query ranking internals.
- Examples
- Uses movie-search text, ts_debug output, numeric and word token types, and English stemming examples.
- What you can do after
- Diagnose tokenization and normalization with PostgreSQL’s ts_debug function and interpret its parser output.
Transfer Learning for Natural Language Processing
The process of dividing text into units such as words, subwords, or characters.
- How it is used
- Uses tokenization for NLP data preprocessing and for preparing BERT inputs for attention inspection.
- How deep it goes
- Shows BERT subword splitting and explains the roles of [CLS] and [SEP] sequence markers.
- Examples
- Encodes a sentence about cells and a cell phone, splitting “didnt” into “didn” and “##t.”
- What you can do after
- Apply it with the Transformers tokenizer to encode text, add special tokens, and map token IDs back to token strings.
Read how Transfer Learning for Natural Language Processing explains tokenization →
Vibe Engineering
The process of converting text into discrete tokens suitable for computational processing.
- How it is used
- Converts raw medical descriptions into fixed numeric inputs for deep-learning classification.
- How deep it goes
- Explains vocabulary mapping, normalization, padding, truncation, and input sizing, but not tokenizer implementation internals.
- Examples
- Uses medicine descriptions, binary word vectors, an Aspirin integer ID, a 20,000-word vocabulary, and 180-position sequences.
- What you can do after
- Apply it via TensorFlow Keras Tokenizer to prepare normalized, vocabulary-bounded, padded text inputs.
Server-Side WebAssembly
Tokenization is the process of dividing text into discrete tokens suitable for processing by a machine-learning model.
- How it is used
- Uses tokenization to prepare text for machine-learning processing inside a Rust WebAssembly guest.
- How deep it goes
- Configures a tokenization library for Wasm but stops short of explaining tokenization algorithms or token-processing internals.
- Examples
- Demonstrates tokenization through a Cargo.toml dependency configured with tokenizers' unstable_wasm feature.
- What you can do after
- Apply tokenization via the Rust tokenizers library in a Wasm guest configured for machine-learning workloads.
Understanding AI
Tokenization is the process of splitting text into reusable units called tokens for language-model processing.
- How it is used
- Explains how language becomes computable input before model processing.
- How deep it goes
- Explains token boundaries, numerical IDs, and vocabulary rationale, but does not detail the tokenization algorithm’s implementation.
- Examples
- Uses “unbelievable,” a full sentence, typos, other languages, code, and an online tokenizer.
- What you can do after
- Operate OpenAI’s online tokenizer to inspect token boundaries and token counts.
Build a Reasoning Model (From Scratch)
The process of rendering text into a model-ready format and splitting it into tokens/IDs for training.
- How it is used
- Prepares repeatedly reused distillation examples before multi-epoch training.
- How deep it goes
- Explains encoding, concatenation, boundary recording, inspection, and filtering, but stops short of tokenizer internals.
- Examples
- Uses a decoded math problem with a reasoning trace, boxed answer, chat markers, and an end-of-sequence token.
- What you can do after
- Builds tokenized training examples with encoded prompts, answers, sequence IDs, prompt lengths, and validation checks.
Read how Build a Reasoning Model (From Scratch) explains tokenization →
Also covered in
- AI Agents in Action
- Transformers in Action
- Regular Expression Puzzles and AI Coding Assistants
- Grokking Machine Learning, Second EditionIn early access (MEAP)
- The AI Pocket Book
- Domain-Specific Small Language Models
- The Quick Python Book, Fourth Edition
- Deep Learning with PyTorch, Second Edition
- Build Applications with Local AI Models on a Mac
- Build a DeepSeek Model (From Scratch)In early access (MEAP)
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
Related concepts
- tokenproduces · A token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- token idproduces · A token ID is an integer identifier assigned to a token in a vocabulary for representing it numerically.
- language modelused by · A language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- splithas part · Split is an operation or function that divides a string or sequence into parts according to a separator, pattern, or other rule.
- standardhas part · Standardization is the establishment or adoption of common rules, formats, specifications, or practices to promote consistency across data, processes, or systems.
- indexinghas part · Indexing is the process of assigning or using positions, keys, or lookup structures to identify and access elements or records.
Questions about tokenization
What is tokenization?
Tokenization is the process of dividing text into discrete units called tokens for computational processing.
Which Manning books explain tokenization?
Learn Generative AI with PyTorch, LLMs in Production, How Large Language Models Work, Deep Learning with R, Third Edition, Software Testing with Generative AI, Deep Learning with Python, Third Edition, Timeless Algorithms, The Foundational Ideas, Data Without Labels and 28 more.
What should I know before learning tokenization?
Manning books that teach tokenization usually introduce language model, token, natural language processing and embedding first.
What are the parts of tokenization?
Tokenization includes split, standard and indexing.
What is tokenization also called?
Tokenization is also called tokenize or tokenizing.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



































