Tokenizer
A tokenizer is a text-processing component or algorithm that segments text into discrete tokens and may map them to identifiers for computational processing.
Natural Language Processing in Action, Second Edition
A tokenizer is a text-processing component that divides a character sequence into tokens such as words, subwords, or punctuation.
- How it is used
- Turns unstructured text into countable units for numerical representations used in search, keyword detection, and text comparison.
- How deep it goes
- Works from whitespace splitting through regular-expression rules, spaCy outputs, and speed benchmarks, without specifying a universally correct tokenization.
- Examples
- Tests a quotation with `str.split`, applies a punctuation-aware regular expression, and compares spaCy pipeline timings.
- What you can do after
- Build it from scratch with Python splitting or regular expressions, then evaluate whether spaCy’s additional features justify their runtime.
Read how Natural Language Processing in Action, Second Edition explains tokenizer →
LLMs in Production
A tokenizer is a text-processing component that divides input text into discrete tokens according to specified rules.
- How it is used
- Prepares text as token indices for language-model training and makes the choice of visible text units part of model design.
- How deep it goes
- Compares word, character, and subword approaches, then demonstrates tokenizer training and model configuration without deriving tokenization algorithms mathematically.
- Examples
- Uses split-word examples, a code-completion failure, subword training code, and a Llama tokenizer with an added padding token.
- What you can do after
- Apply it via Hugging Face tokenizers to train a subword tokenizer or configure a model tokenizer for padded batches.
Deep Learning with R, Third Edition
A tokenizer is a text-processing component that converts input text into a sequence of discrete tokens, often assigning each token an integer index.
- How it is used
- Converts text into token IDs for text classification, pretrained Transformers, and language-model training.
- How deep it goes
- Builds character, word, and byte-pair tokenizers, then uses packaged tokenizers with pretrained models.
- Examples
- Uses a fox sentence, Moby Dick vocabulary counts, RoBERTa token IDs, and a SentencePiece round trip.
- What you can do after
- Build it from scratch with character, word, or byte-pair splitting; apply it via KerasHub when using a matching pretrained model.
Read how Deep Learning with R, Third Edition explains tokenizer →
How Large Language Models Work
A tokenizer is a software component or algorithm that converts text into tokens according to a vocabulary and tokenization procedure.
- How it is used
- Converts natural-language or programming-code input into numeric units that language models can process.
- How deep it goes
- Traces normalization, segmentation, BPE vocabulary construction, numeric encoding, and practical limitations without detailing model internals.
- Examples
- Uses running versus runnin, loquacious, digit-separated numbers, homoglyphs, multilingual text, and Python–Java code comparisons.
- What you can do after
- Apply the OpenAI tokenizer to inspect token boundaries, compare casing and phrasing, and anticipate counting, testing, cost, and arithmetic effects.
Read how How Large Language Models Work explains tokenizer →
Software Testing with Generative AI
A tokenizer is a tool that converts text into tokens and commonly maps those tokens to numeric identifiers.
- How it is used
- Supports prompt design and dataset preprocessing for language-model use and fine-tuning.
- How deep it goes
- Explains token boundaries, integer identifiers, reconstruction, model alignment, and context-limit risks, but does not detail tokenizer algorithms.
- Examples
- Uses a populated-cities prompt, integer token lists, token-preview experimentation, and Axolotl preprocessing.
- What you can do after
- Apply tokenizer previews and preprocessing to inspect tokenization, select a compatible tokenizer, and avoid context overflow.
Read how Software Testing with Generative AI explains tokenizer →
Build a Reasoning Model (From Scratch)
A component that splits text into token IDs for model input, with this section focusing on the Qwen3 reasoning variant that supports reasoning tags.
- How it is used
- Prepares prompts, reasoning traces, training answers, and generated responses for LLM processing.
- How deep it goes
- Explains loading, encoding, decoding, subword segmentation, chat wrapping, and training-sequence assembly, but not tokenizer algorithm internals.
- Examples
- Demonstrates Qwen3 prompt round trips, per-token ID inspection, chat-wrapped reasoning prompts, and concatenated training sequences.
- What you can do after
- Apply Qwen3Tokenizer to encode prompts, decode outputs, inspect subwords, and assemble reasoning-model training sequences.
Read how Build a Reasoning Model (From Scratch) explains tokenizer →
Build a Text-to-Image Generator (from Scratch)
A tokenizer is a text-processing component that segments a character string into discrete tokens or token identifiers.
- How it is used
- Prepares German-English sentences and image-generation prompts for neural models.
- How deep it goes
- Explains tokenization, vocabulary mapping, special tokens, padding, and encoder handoff, but defers tokenizer training internals.
- Examples
- Uses German-English translation sentences, “a lion” and “a tiger,” and “panda with top hat reading a book.”
- What you can do after
- Builds token-to-index preprocessing and applies pretrained tokenizers to translation and image-generation prompts.
Read how Build a Text-to-Image Generator (from Scratch) explains tokenizer →
Elasticsearch in Action, Second Edition
A tokenizer is a text-analysis component that divides a character stream into discrete tokens according to specified rules.
- How it is used
- Splits full-text fields and search queries into terms for Elasticsearch analysis and indexing.
- How deep it goes
- Explains splitting rules, analyzer placement, built-in options, and custom-analyzer testing, while deferring exhaustive tokenizer coverage.
- Examples
- Demonstrates movie-review splitting, hyphenated text, HTML stripping, uppercasing, and preservation of URLs, emails, or paths.
- What you can do after
- Apply it via Elasticsearch analyzers and the _analyze API to inspect token output and configure custom analysis.
Read how Elasticsearch in Action, Second Edition explains tokenizer →
Server-Side WebAssembly
A tokenizer is a text-processing component that converts text into discrete tokens and can reconstruct text from token sequences.
- How it is used
- Bridges story prompts and generated model output in a Wasm ML component.
- How deep it goes
- Works through loading, encoding IDs, EOS lookup, iterative decoding, and stopping checks without detailing tokenizer training or vocabulary construction.
- Examples
- Uses a tokenizer.json fixture and the prompt “Once upon a time” to generate a story token by token.
- What you can do after
- Apply it via the tokenizers library to encode prompts and decode sampled IDs in a Wasm story generator.
Deep Learning with Python, Third Edition
A component that converts text into a sequence of integer token IDs using a vocabulary and tokenization rules.
- How it is used
- Prepares text for vocabulary lookup, neural models, and pretrained Transformer inputs.
- How deep it goes
- Explains standardization, splitting, vocabulary indexing, unknown tokens, and pretrained-tokenizer compatibility, but not tokenizer training internals.
- Examples
- Uses character and word splitting, vocabulary lookup with [UNK], and RoBERTa tokenization through KerasHub.
- What you can do after
- Apply it via KerasHub by loading a matching RoBERTa tokenizer and converting text into integer arrays.
Read how Deep Learning with Python, Third Edition explains tokenizer →
AI-Powered Search
A tokenizer is a text-processing component that segments text into tokens according to a model's tokenization scheme.
- How it is used
- Prepares documents, queries, and question-answering examples for Transformer encoding or training.
- How deep it goes
- Explains word-piece segmentation, special tokens, sliding windows, overlap, truncation, and padding without detailing tokenizer internals.
- Examples
- Uses “It’s” word pieces, [CLS] and [SEP], and 384-token question-context windows overlapping by 128 tokens.
- What you can do after
- Apply tokenization with a pretrained RoBERTa tokenizer to prepare fixed-shape inputs for model training and evaluation.
Starting Data Analytics with Generative AI and Python
A tokenizer is a software component that segments text into discrete tokens according to specified processing rules, often for a particular language.
- How it is used
- Supports language-aware text preprocessing before frequency analysis of customer reviews.
- How deep it goes
- Explains tokenization’s role and demonstrates a language-specific correction, but does not develop tokenizer internals.
- Examples
- Uses Portuguese customer reviews in review_comment_message and contrasts default and language-aware NLTK tokenization.
- What you can do after
- Apply it via NLTK to tokenize Portuguese reviews and diagnose language-default mistakes in generated code.
Read how Starting Data Analytics with Generative AI and Python explains tokenizer →
Data Analysis with LLMs
A tokenizer is a software component that transforms text into the token identifiers used by a language model.
- How it is used
- Uses tokenizers to identify token IDs required for logit-bias configuration in language-model classification.
- How deep it goes
- Explains the transformation and demonstrates ID lookup, but stops short of tokenizer implementation internals.
- Examples
- Demonstrates GPT-3.5 class labels pos and neg, IDs 981 and 29875, and biased API requests.
- What you can do after
- Operate the GPT tokenizer interface to find token IDs and configure targeted model generation.
Domain-Specific Small Language Models
A model-specific component that converts concatenated instruction/output text into token sequences for fine-tuning.
- How it is used
- Supports fine-tuning and inference by turning task text into model-compatible token sequences and numerical inputs.
- How deep it goes
- Explains token types, model-specific formats, special tokens, padding, truncation, and IDs, but not tokenizer construction internals.
- Examples
- Demonstrates BERT sentiment inputs, question-answering preprocessing, GPT-Neo prompts, padding, truncation, and answer decoding.
- What you can do after
- Apply it via Hugging Face Transformers to prepare padded or truncated inputs and decode model outputs.
Read how Domain-Specific Small Language Models explains tokenizer →
Learn Generative AI with PyTorch
A text-processing component that splits text into tokens and converts text to a discrete token sequence for modeling.
- How it is used
- Prepares English and French translation text for Transformer modeling.
- How deep it goes
- Explains subword splitting, boundary markers, vocabulary construction, index mapping, and reconstruction.
- Examples
- Uses English–French sentence pairs, subword outputs, special tokens, frequency counts, indexes, and reconstructed phrases.
- What you can do after
- Apply it via Hugging Face Transformers and custom dictionaries to convert multilingual sentences into model indexes.
Read how Learn Generative AI with PyTorch explains tokenizer →
Generative AI in Action
A tokenizer is a component that segments or encodes text into discrete units for processing by a language model.
- How it is used
- Supports prompt sizing and model selection by exposing token counts that affect context quality, cost, and performance.
- How deep it goes
- Explains token counting, BPE properties, and model-specific encodings, while stopping short of tokenizer training internals.
- Examples
- Uses tiktoken code to count tokens for a prompt and compare cl100k_base, p50k_base, and r50k_base encodings.
- What you can do after
- Operate tokenization via tiktoken to count, encode, decode, and select model-compatible encodings.
Related concepts
- tokenproduces · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
- tokenizationimplements · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
- token idproduces · A token ID is a numeric identifier assigned to a token in a vocabulary so it can be represented and processed numerically.
- text analyzerpart of · A text analyzer is a component that transforms text into terms or other linguistic representations for indexing and searching.
- spaCyused by · spaCy is an open-source Python library for natural language processing, providing tools and pretrained pipelines for tasks such as tokenization and linguistic analysis.
- standard tokenizerincludes · A standard tokenizer is a built-in Lucene text-analysis component that divides text into tokens according to predefined standard rules.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.