liveBook · Manning

Tokenizer

A tokenizer is a text-processing component or procedure that divides text into discrete tokens and may map them to identifiers according to specified rules or a vocabulary.

How 19 Manning books explain it

Natural Language Processing in Action, Second Edition

A tokenizer is a text-processing component that divides a character sequence into tokens such as words, subwords, or punctuation.

How it is used
Uses tokenization to turn text into discrete units for numerical representations, search, and classification.
How deep it goes
Works through whitespace, regular expression, and spaCy tokenization, including boundary choices, accuracy, and speed tradeoffs.
Examples
Demonstrates splitting a literary quote, matching contractions with a regular expression, and tokenizing the same passage with spaCy.
What you can do after
Apply it via Python or spaCy, inspect token boundaries, and compare tokenizer choices against task accuracy and latency.

Read how Natural Language Processing in Action, Second Edition explains tokenizer →

LLMs in Production

A tokenizer is a text-processing component that divides input text into discrete tokens according to specified rules.

How it is used
Prepares text as numerical input for language-model training, fine-tuning, and deployment.
How deep it goes
Contrasts word, character, and subword approaches and demonstrates custom training and tokenizer configuration.
Examples
Uses code completion, arithmetic tokenization, and a Llama tokenizer setup to illustrate practical choices.
What you can do after
Apply it via Hugging Face tokenizers to train a BPE vocabulary or configure a tokenizer with a padding token.

Read how LLMs in Production explains tokenizer →

Deep Learning with R, Third Edition

A tokenizer is a text-processing component that converts input text into a sequence of discrete tokens, often assigning each token an integer index.

How it is used
Prepares text as integer sequences for classification, pretrained Transformers, and text generation.
How deep it goes
Builds character, word, and byte-pair tokenizers, including vocabulary computation, indexing, and detokenization.
Examples
Compares tokenizations of Moby Dick and demonstrates SentencePiece on a short fox sentence.
What you can do after
Build tokenizers from scratch and apply pretrained tokenizers via KerasHub and SentencePiece.

Read how Deep Learning with R, Third Edition explains tokenizer →

How Large Language Models Work

A tokenizer is a software component or algorithm that converts text into tokens according to a vocabulary and tokenization procedure.

How it is used
Converts natural-language or programming-code input into numeric units that language models can process.
How deep it goes
Traces normalization, segmentation, BPE vocabulary construction, numeric encoding, and practical limitations without detailing model internals.
Examples
Uses running versus runnin, loquacious, digit-separated numbers, homoglyphs, multilingual text, and Python–Java code comparisons.
What you can do after
Apply the OpenAI tokenizer to inspect token boundaries, compare casing and phrasing, and anticipate counting, testing, cost, and arithmetic effects.

Read how How Large Language Models Work explains tokenizer →

Software Testing with Generative AI

A tokenizer is a tool that converts text into tokens and commonly maps those tokens to numeric identifiers.

How it is used
Supports prompt design and dataset preprocessing for language-model use and fine-tuning.
How deep it goes
Explains token boundaries, integer identifiers, reconstruction, model alignment, and context-limit risks, but does not detail tokenizer algorithms.
Examples
Uses a populated-cities prompt, integer token lists, token-preview experimentation, and Axolotl preprocessing.
What you can do after
Apply tokenizer previews and preprocessing to inspect tokenization, select a compatible tokenizer, and avoid context overflow.

Read how Software Testing with Generative AI explains tokenizer →

Build a Reasoning Model (From Scratch)

A component that splits text into token IDs for model input, with this section focusing on the Qwen3 reasoning variant that supports reasoning tags.

How it is used
Prepares prompts, reasoning traces, training answers, and generated responses for LLM processing.
How deep it goes
Explains loading, encoding, decoding, subword segmentation, chat wrapping, and training-sequence assembly, but not tokenizer algorithm internals.
Examples
Demonstrates Qwen3 prompt round trips, per-token ID inspection, chat-wrapped reasoning prompts, and concatenated training sequences.
What you can do after
Apply Qwen3Tokenizer to encode prompts, decode outputs, inspect subwords, and assemble reasoning-model training sequences.

Read how Build a Reasoning Model (From Scratch) explains tokenizer →

Build a Text-to-Image Generator (from Scratch)

A tokenizer is a text-processing component that segments a character string into discrete tokens or token identifiers.

How it is used
Prepares German-English sentences and image-generation prompts for neural models.
How deep it goes
Explains tokenization, vocabulary mapping, special tokens, padding, and encoder handoff, but defers tokenizer training internals.
Examples
Uses German-English translation sentences, “a lion” and “a tiger,” and “panda with top hat reading a book.”
What you can do after
Builds token-to-index preprocessing and applies pretrained tokenizers to translation and image-generation prompts.

Read how Build a Text-to-Image Generator (from Scratch) explains tokenizer →

Elasticsearch in Action, Second Edition

A tokenizer is a text-analysis component that divides a character stream into discrete tokens according to specified rules.

How it is used
Splits full-text fields and search queries into terms for Elasticsearch analysis and indexing.
How deep it goes
Explains splitting rules, analyzer placement, built-in options, and custom-analyzer testing, while deferring exhaustive tokenizer coverage.
Examples
Demonstrates movie-review splitting, hyphenated text, HTML stripping, uppercasing, and preservation of URLs, emails, or paths.
What you can do after
Apply it via Elasticsearch analyzers and the _analyze API to inspect token output and configure custom analysis.

Read how Elasticsearch in Action, Second Edition explains tokenizer →

Retrieval Augmented Generation, The Foundational Ideas

A tokenizer converts text into token identifiers and related tensors that a neural language model can process.

How it is used
Converts batched question-and-passage strings into model inputs for independent FiD encoding, then converts generated IDs into an answer.
How deep it goes
Shows tokenizer calls and the resulting tensors in a worked FiD implementation, without explaining tokenization algorithms.
Examples
Uses a Hugging Face tokenizer with longest-batch padding and truncation for retrieved passages, plus decoding of generated IDs.
What you can do after
Apply it via Hugging Face Transformers to prepare FiD encoder inputs and decode a generated answer.

Read how Retrieval Augmented Generation, The Foundational Ideas explains tokenizer →

Server-Side WebAssembly

A tokenizer is a text-processing component that converts text into discrete tokens and can reconstruct text from token sequences.

How it is used
Bridges story prompts and generated model output in a Wasm ML component.
How deep it goes
Works through loading, encoding IDs, EOS lookup, iterative decoding, and stopping checks without detailing tokenizer training or vocabulary construction.
Examples
Uses a tokenizer.json fixture and the prompt “Once upon a time” to generate a story token by token.
What you can do after
Apply it via the tokenizers library to encode prompts and decode sampled IDs in a Wasm story generator.

Read how Server-Side WebAssembly explains tokenizer →

Deep Learning with Python, Third Edition

A component that converts text into a sequence of integer token IDs using a vocabulary and tokenization rules.

How it is used
Prepares text for vocabulary lookup, neural models, and pretrained Transformer inputs.
How deep it goes
Explains standardization, splitting, vocabulary indexing, unknown tokens, and pretrained-tokenizer compatibility, but not tokenizer training internals.
Examples
Uses character and word splitting, vocabulary lookup with [UNK], and RoBERTa tokenization through KerasHub.
What you can do after
Apply it via KerasHub by loading a matching RoBERTa tokenizer and converting text into integer arrays.

Read how Deep Learning with Python, Third Edition explains tokenizer →

AI-Powered Search

A tokenizer is a text-processing component that segments text into tokens according to a model's tokenization scheme.

How it is used
Prepares documents, queries, and question-answering examples for Transformer encoding or training.
How deep it goes
Explains word-piece segmentation, special tokens, sliding windows, overlap, truncation, and padding without detailing tokenizer internals.
Examples
Uses “It’s” word pieces, [CLS] and [SEP], and 384-token question-context windows overlapping by 128 tokens.
What you can do after
Apply tokenization with a pretrained RoBERTa tokenizer to prepare fixed-shape inputs for model training and evaluation.

Read how AI-Powered Search explains tokenizer →

Starting Data Analytics with Generative AI and Python

A tokenizer is a software component that segments text into discrete tokens according to specified processing rules, often for a particular language.

How it is used
Supports language-aware text preprocessing before frequency analysis of customer reviews.
How deep it goes
Explains tokenization’s role and demonstrates a language-specific correction, but does not develop tokenizer internals.
Examples
Uses Portuguese customer reviews in review_comment_message and contrasts default and language-aware NLTK tokenization.
What you can do after
Apply it via NLTK to tokenize Portuguese reviews and diagnose language-default mistakes in generated code.

Read how Starting Data Analytics with Generative AI and Python explains tokenizer →

Data Analysis with LLMs

A tokenizer is a software component that transforms text into the token identifiers used by a language model.

How it is used
Uses tokenizers to identify token IDs required for logit-bias configuration in language-model classification.
How deep it goes
Explains the transformation and demonstrates ID lookup, but stops short of tokenizer implementation internals.
Examples
Demonstrates GPT-3.5 class labels pos and neg, IDs 981 and 29875, and biased API requests.
What you can do after
Operate the GPT tokenizer interface to find token IDs and configure targeted model generation.

Read how Data Analysis with LLMs explains tokenizer →

Domain-Specific Small Language Models

A model-specific component that converts concatenated instruction/output text into token sequences for fine-tuning.

How it is used
Converts text into model-ready token sequences for training and inference.
How deep it goes
Explains token units, model-specific formats, special tokens, padding, and truncation, without detailing tokenizer internals.
Examples
Uses BERT sentiment and question-answering inputs, plus GPT-Neo prompts and generated text.
What you can do after
Apply it via Hugging Face Transformers to encode training examples and prompts and decode generated tokens.

Read how Domain-Specific Small Language Models explains tokenizer →

Learn Generative AI with PyTorch

A text-processing component that splits text into tokens and converts text to a discrete token sequence for modeling.

How it is used
Prepares English and French translation text for Transformer modeling.
How deep it goes
Explains subword splitting, boundary markers, vocabulary construction, index mapping, and reconstruction.
Examples
Uses English–French sentence pairs, subword outputs, special tokens, frequency counts, indexes, and reconstructed phrases.
What you can do after
Apply it via Hugging Face Transformers and custom dictionaries to convert multilingual sentences into model indexes.

Read how Learn Generative AI with PyTorch explains tokenizer →

Transfer Learning for Natural Language Processing

A tokenizer is a software component that divides text into tokens and may convert them into model-ready representations.

How it is used
Prepares text as model input by splitting it into tokens and mapping them to vocabulary IDs for BERT workflows.
How deep it goes
Demonstrates pretrained and from-scratch setup, subword splitting, special tokens, vocabulary saving, and sequence-length configuration.
Examples
Uses an English word split into “didn” and “##t,” and trains a WordPiece tokenizer on monolingual Twi text.
What you can do after
Apply Transformers tokenizers to encode BERT inputs, or train and save a Twi WordPiece vocabulary with Tokenizers.

Read how Transfer Learning for Natural Language Processing explains tokenizer →

Generative AI in Action

A tokenizer is a component that segments or encodes text into discrete units for processing by a language model.

How it is used
Supports prompt sizing and model selection by exposing token counts that affect context quality, cost, and performance.
How deep it goes
Explains token counting, BPE properties, and model-specific encodings, while stopping short of tokenizer training internals.
Examples
Uses tiktoken code to count tokens for a prompt and compare cl100k_base, p50k_base, and r50k_base encodings.
What you can do after
Operate tokenization via tiktoken to count, encode, decode, and select model-compatible encodings.

Read how Generative AI in Action explains tokenizer →

Machines that Think

A tokenizer is a software component that segments text into discrete units for computational language processing.

How it is used
Represents text as model-usable units while shaping multilingual processing and computational efficiency.
How deep it goes
Compares learned segmentation approaches and explains how token granularity affects nuance, sequence length, memory, and context capacity.
Examples
Contrasts whitespace-free Japanese, Chinese, and Thai with multilingual processing that includes English, Korean, and Arabic.
What you can do after
Evaluate how segmentation choices trade preserved detail against sequence length and computational efficiency.

Read how Machines that Think explains tokenizer →

Also covered in

What to know first

Related concepts

Questions about tokenizer

What is tokenizer?

A tokenizer is a text-processing component or procedure that divides text into discrete tokens and may map them to identifiers according to specified rules or a vocabulary.

Which Manning books explain tokenizer?

Natural Language Processing in Action, Second Edition, LLMs in Production, Deep Learning with R, Third Edition, How Large Language Models Work, Software Testing with Generative AI, Build a Reasoning Model (From Scratch), Build a Text-to-Image Generator (from Scratch), Elasticsearch in Action, Second Edition and 11 more.

What should I know before learning tokenizer?

Manning books that teach tokenizer usually introduce language model, token, vocabulary and fine-tuning first.

What is tokenizer part of?

Tokenizer is part of text analyzer. A text analyzer is a component that transforms text into terms or other linguistic representations for indexing and searching.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.