Tokenization
Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
Learn Generative AI with PyTorch
The process of splitting text into discrete units such as characters, words, or subwords for modeling.
- How it is used
- Prepares ordered text or music events as indexed sequences for next-token prediction in LSTMs and Transformers.
- How deep it goes
- Works through word-token cleaning and indexing, demonstrates pretrained subword tokenization, and defers the detailed MIDI-to-event conversion.
- Examples
- Uses short English phrases, bilingual translation pairs, three novels, and MIDI performance events to show token choices and indexed sequences.
- What you can do after
- Build it from scratch for word-level text, or apply it via XLMTokenizer to prepare English–French translation pairs.
Read how Learn Generative AI with PyTorch explains tokenization →
LLMs in Production
Tokenization is the process of dividing text into discrete units that can be represented and processed by a language model.
- How it is used
- Prepares raw text as numerical input for language-model training and inference.
- How deep it goes
- Explains token units, unknown-token failures, strategy trade-offs, and tokenizer configuration; does not derive a tokenization algorithm.
- Examples
- Uses a code-completion failure, whitespace-based word counts, arithmetic token splits, and a Llama 3 tokenizer setup.
- What you can do after
- Operate/configure a tokenizer by inspecting task-specific splits, choosing whether to reuse or adapt it, and adding padding when needed.
How Large Language Models Work
Tokenization is the process of converting an input into discrete units or a sequence of numeric representations for model processing.
- How it is used
- Prepares text as numeric model input and shapes which content an LLM can represent efficiently.
- How deep it goes
- Explains normalization, segmentation, numeric mapping, vocabulary trade-offs, BPE, and capability risks without dense implementation detail.
- Examples
- Uses whitespace splitting, BPE pair merging, running versus runnin, digit-level arithmetic, multilingual token costs, and code examples.
- What you can do after
- Evaluate tokenizer effects on vocabulary, cost, language coverage, arithmetic, code, and application-specific representation.
Read how How Large Language Models Work explains tokenization →
Deep Learning with R, Third Edition
Word tokenization is the division of text into tokens corresponding primarily to words.
- How it is used
- Converts raw text into token IDs for text classification and language-model training.
- How deep it goes
- Works through standardization, splitting, vocabulary construction, indexing, and unknown tokens, then compares character, word, and subword choices.
- Examples
- Uses a sample sentence, Moby Dick vocabulary counts, IMDb reviews, and next-character prediction on Shakespeare text.
- What you can do after
- Build it from scratch with R vocabulary functions, or apply it via Keras TextVectorization to classification and language-model datasets.
Read how Deep Learning with R, Third Edition explains tokenization →
Software Testing with Generative AI
Tokenization is the process of converting text into discrete tokens that a language model can process as numeric identifiers.
- How it is used
- Supports prompt design and fine-tuning data preparation by translating text into the representation language models process.
- How deep it goes
- Explains token splitting, integer mapping, tokenizer alignment, and context-length truncation, but does not detail tokenizer algorithms.
- Examples
- Uses a city-list prompt, integer token sequences, and fine-tuning examples involving test-code instructions and outputs.
- What you can do after
- Evaluate prompt and training-data limits by estimating token boundaries, checking tokenizer alignment, and identifying context-length truncation risks.
Read how Software Testing with Generative AI explains tokenization →
Deep Learning with Python, Third Edition
The process of converting text into a sequence of discrete units such as words, characters, or punctuation marks.
- How it is used
- Converts raw text into numeric inputs for text classifiers and pretrained language models.
- How deep it goes
- Works through standardization, regex splitting, vocabulary construction, unknown-token indexing, and character–word trade-offs; introduces subwords without an implementation here.
- Examples
- Uses a regex-split sentence, character and word tokenizers tested on Moby Dick, and packed IMDb reviews.
- What you can do after
- Build it from scratch with a counted vocabulary, or apply it via Keras TextVectorization and a RoBERTa tokenizer.
Read how Deep Learning with Python, Third Edition explains tokenization →
Data Without Labels
Tokenization is the process of splitting text into discrete units such as words or punctuation marks called tokens.
- How it is used
- Prepares text as tokens for vocabularies and downstream text representations.
- How deep it goes
- Defines word, subword, and character tokens, surveys boundary methods, and demonstrates NLTK calls without detailing tokenizer internals.
- Examples
- Uses a sentence split into tokens, Python cleaning snippets, and tokenized customer reviews in a sentiment workflow.
- What you can do after
- Apply it via NLTK to inspect tokens and prepare cleaned text for further processing.
Hugging Face in Action
A tokenization strategy is a rule or method for dividing text into model-processing units.
- How it is used
- Prepares text and datasets as numerical inputs for NLP models, inference, visualization, and fine-tuning.
- How deep it goes
- Explains three tokenization strategies and dataset fields, then demonstrates batching, padding, truncation, and token-ID inspection.
- Examples
- Uses BERT subwords, word-level sentences, multilingual character examples, sentiment inputs, and mapped training datasets.
- What you can do after
- Apply it via Hugging Face AutoTokenizer to tokenize text, inspect tokens and IDs, and prepare padded datasets for model use.
Time Series Forecasting Using Foundation Models
Tokenization is the process of converting input data into discrete units suitable for sequence-model processing.
- How it is used
- Converts time-series values, lags, and calendar information into inputs that decoder or language models can process for forecasting.
- How deep it goes
- Explains feature construction, mean scaling, quantization, reversal, and precision trade-offs, while deferring implementation code and formal algorithm details.
- Examples
- Uses hourly calendar features, store-sales and road-traffic scales, mean-scaled series, and comparisons between 100-bin and five-bin vocabularies.
- What you can do after
- Apply it via Lag-Llama or Chronos by constructing features or scaled bins, then restoring Chronos outputs through dequantization and unscaling.
Read how Time Series Forecasting Using Foundation Models explains tokenization →
Natural Language Processing in Action, Second Edition
Tokenization is the process of dividing a character sequence into discrete lexical units such as words or punctuation.
- How it is used
- Turns unstructured text into numerical inputs for NLP pipelines, search, classification, information retrieval, and conversational systems.
- How deep it goes
- Traces segmentation from documents to tokens, compares word and subword units, and reaches vectorization and preprocessing tradeoffs.
- Examples
- Uses greetings such as Hi and Hello, confusable strings such as Hiking and Hell, emojis, logograms, and subword encodings.
- What you can do after
- Apply it via a configured tokenizer to represent text, detect keywords, support search, and prepare tokens for numerical encoding.
Read how Natural Language Processing in Action, Second Edition explains tokenization →
Build a Text-to-Image Generator (from Scratch)
Tokenization is the process of splitting text into discrete tokens such as words, punctuation marks, or subwords.
- How it is used
- Converts raw language into discrete units and numerical inputs for translation, transformers, and text-to-image generation.
- How deep it goes
- Explains splitting, vocabulary construction, special indices, padding, and embedding handoff, but defers tokenizer internals.
- Examples
- Uses German-English translation phrases, “Wie geht es dir,” animal prompts, and “panda with top hat reading a book.”
- What you can do after
- Apply it via spaCy or Transformers to tokenize prompts, manage vocabulary indices, and prepare model-ready embeddings.
Read how Build a Text-to-Image Generator (from Scratch) explains tokenization →
MongoDB 8.0 in Action, Third Edition
Tokenization is the process of dividing a stream of text into smaller units called tokens.
- How it is used
- Prepares document text and queries as searchable terms for full-text search.
- How deep it goes
- Explains splitting, normalization, filters, and analyzer selection, but leaves custom-analyzer construction to external documentation.
- Examples
- Uses two inspection documents, their shared indexed terms, a matching query, and a table of built-in analyzers.
- What you can do after
- Operate/configure it in Atlas Search by selecting indexing and query analyzers or defining a custom analyzer.
Read how MongoDB 8.0 in Action, Third Edition explains tokenization →
Build a Large Language Model (From Scratch)
The process of splitting raw text into tokens for further processing by a language model.
- How it is used
- Prepares text for LLM pretraining by producing tokens that can become vector inputs.
- How deep it goes
- Explains regular-expression tokenization, whitespace choices, and downstream encoding, while deferring detailed byte pair encoding implementation.
- Examples
- Demonstrates Python regular-expression splits, punctuation cleanup, whitespace filtering, and preprocessing a short story.
- What you can do after
- Build it from scratch with Python regular expressions and produce cleaned word-and-punctuation token lists for later encoding.
Read how Build a Large Language Model (From Scratch) explains tokenization →
AI-Powered Search
Tokenization is the process of segmenting text into units such as words, subwords, or punctuation symbols.
- How it is used
- Supports query interpretation, spelling correction, semantic encoding, and question-answering model preparation.
- How deep it goes
- Explains filtering, whole-query alternatives, model vocabulary, sliding windows, overlap, and padding without detailing tokenizer internals.
- Examples
- Uses filtered query logs, product-name misspellings, and question-answering contexts divided into overlapping fixed-size token windows.
- What you can do after
- Apply tokenization with NLTK or a Transformers tokenizer, while choosing between term-level and whole-query processing.
Elasticsearch in Action, Second Edition
Tokenization is the process of splitting a character stream into discrete tokens according to rules such as whitespace, delimiters, or character patterns.
- How it is used
- Prepares analyzed text fields for query matching by converting character streams into searchable tokens.
- How deep it goes
- Defines rule-based splitting and variants, but defers synonym handling, stemming, and other transformations to normalization.
- Examples
- Uses synopsis text, whitespace-separated words, movie-review punctuation, hyphenated terms, and alphanumeric strings as examples.
- What you can do after
- Evaluate tokenizer choices for searchable text and distinguish splitting from subsequent normalization.
Read how Elasticsearch in Action, Second Edition explains tokenization →
Build AI-Enhanced Web Apps
Tokenization is the conversion of text or other input into a sequence of tokens or numerical token identifiers.
- How it is used
- Supports efficient communication between prompts and LLMs while helping developers estimate interaction costs.
- How deep it goes
- Explains the input-output pipeline, numerical IDs, billing, and JavaScript token counting, but stops short of tokenizer internals.
- Examples
- Uses a translation prompt, a system message, token-counting code, and an end-to-end processing diagram.
- What you can do after
- Apply it via @dqbd/tiktoken to count model-specific prompt and system-message tokens.
Generative AI in Action
Tokenization is the process of converting text into a sequence of discrete tokens for language-model processing.
- How it is used
- Connects LLM processing and enterprise search design to token boundaries, context, cost, latency, and field behavior.
- How deep it goes
- Defines tokenization, explains iterative next-token generation, and stops short of algorithm internals.
- Examples
- Uses an eight-token dog sentence and contrasts Redis TagField exact matching with TextField tokenized searching.
- What you can do after
- Evaluate token costs and configure Redis fields for exact-match or tokenized search behavior.
Starting Data Analytics with Generative AI and Python
An algorithm that divides text into the token units consumed by a language model.
- How it is used
- Frames tokenization as the practical basis for estimating prompt capacity and managing generative AI context limits.
- How deep it goes
- Defines token units and gives an average ratio, but stops short of explaining algorithm internals or comparing specific methods.
- Examples
- Uses words, word parts, and punctuation marks as illustrative token units.
- What you can do after
- Evaluate when inputs may exceed context limits and recognize truncation through inconsistent responses or forgotten exchanges.
Read how Starting Data Analytics with Generative AI and Python explains tokenization →
AI Agents in Action, Second Edition
Tokenization is the process of converting text into discrete tokens that a language model can process.
- How it is used
- Supports LLM input preparation, cost estimation, context management, and quality control.
- How deep it goes
- Explains token pieces, vocabulary indices, generation, JSON overhead, and measurement, but stops short of tokenizer algorithm internals.
- Examples
- Compares regular text with JSON and recommends measuring counts using tiktoken or Agents SDK telemetry.
- What you can do after
- Apply it via tiktoken or Agents SDK telemetry to measure tokens and manage budgets, context windows, and prompt overhead.
Read how AI Agents in Action, Second Edition explains tokenization →
Grokking AI Algorithms, Second Edition
Tokenization is the process of breaking text into a sequence of discrete units for computational language processing.
- How it is used
- Prepares language-model text by converting it into vocabulary units, token IDs, streams, and context-window-sized batches.
- How deep it goes
- Explains character initialization, end-of-word markers, bigram counts, repeated BPE merges, vocabulary IDs, and batching.
- Examples
- Uses Alice's sentence, the word unbelievable, character markers, bigram tables, token-ID listings, token streams, and Python batching code.
- What you can do after
- Build it from scratch by merging frequent character pairs, assigning token IDs, and slicing token streams into training batches.
Read how Grokking AI Algorithms, Second Edition explains tokenization →
Write Powerful Rust Macros
Lexing is the process of converting raw source text into a sequence of lexical tokens.
- How it is used
- Explains the representation procedural macros consume and produce while positioning lexing before parsing.
- How deep it goes
- Defines tokenization and contrasts it with parsing, but stops short of detailed compiler internals.
- Examples
- Uses `1 + 11` and its TokenStream representation to demonstrate literals and punctuation.
- What you can do after
- Evaluate/understand how Rust represents macro input before parsing and code generation.
Coding with AI
Tokenization is the process of breaking text into smaller units such as words, punctuation marks, or phrases.
- How it is used
- Supports NLP tasks by converting text into units that can be counted, compared, classified, and analyzed.
- How deep it goes
- Defines word-based tokenization and demonstrates preprocessing and counting, while deferring deeper NLP treatment.
- Examples
- Uses a sentence-token list and a truck-frequency example to demonstrate punctuation handling, normalization, and word counting.
- What you can do after
- Apply it in a Python workflow that cleans text, counts word frequencies, and displays the most frequent words.
Just Use Postgres!
Tokenization is the process of splitting text into discrete tokens for linguistic or computational processing.
- How it is used
- Prepares document text for PostgreSQL full-text search by producing searchable tokens and lexemes.
- How deep it goes
- Explains parsing, token types, dictionaries, stemming, and stop-word removal, then stops short of query ranking internals.
- Examples
- Uses movie-search text, ts_debug output, numeric and word token types, and English stemming examples.
- What you can do after
- Diagnose tokenization and normalization with PostgreSQL’s ts_debug function and interpret its parser output.
Server-Side WebAssembly
Tokenization is the process of dividing text into discrete tokens suitable for processing by a machine-learning model.
- How it is used
- Uses tokenization to prepare text for machine-learning processing inside a Rust WebAssembly guest.
- How deep it goes
- Configures a tokenization library for Wasm but stops short of explaining tokenization algorithms or token-processing internals.
- Examples
- Demonstrates tokenization through a Cargo.toml dependency configured with tokenizers' unstable_wasm feature.
- What you can do after
- Apply tokenization via the Rust tokenizers library in a Wasm guest configured for machine-learning workloads.
Build a Reasoning Model (From Scratch)
The process of rendering text into a model-ready format and splitting it into tokens/IDs for training.
- How it is used
- Prepares repeatedly reused distillation examples before multi-epoch training.
- How deep it goes
- Explains encoding, concatenation, boundary recording, inspection, and filtering, but stops short of tokenizer internals.
- Examples
- Uses a decoded math problem with a reasoning trace, boxed answer, chat markers, and an end-of-sequence token.
- What you can do after
- Builds tokenized training examples with encoded prompts, answers, sequence IDs, prompt lengths, and validation checks.
Read how Build a Reasoning Model (From Scratch) explains tokenization →
Related concepts
- tokenproduces · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
- token idproduces · A token ID is a numeric identifier assigned to a token in a vocabulary so it can be represented and processed numerically.
- language modelused by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- standardhas part · A standard is an established rule, specification, or convention that defines a consistent reference for practice, representation, behavior, or evaluation.
- indexinghas part · Indexing is the organization or use of positions, keys, or lookup structures to identify and retrieve selected data or elements.
- splithas part · Split is an operation or function that divides a string or sequence into parts according to a separator, pattern, or other rule.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.