liveBook · Manning

Token

A token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.

How 40 Manning books explain it

How Large Language Models Work

A token is a discrete symbolic unit that represents a subword through a unique numeric identifier for processing by a language model.

How it is used
Uses tokens as the numeric input units through which an LLM represents, learns from, and generates text.
How deep it goes
Explains tokenization through normalization, segmentation, BPE, vocabulary trade-offs, and failure cases, while stopping short of implementation code.
Examples
Uses schoolhouse, thoughtful, running, misspelled drug names, character puzzles, and digit-level arithmetic tokenization.
What you can do after
Evaluate tokenization-related risks in applications involving spelling, characters, mathematics, languages, cost, or domain vocabulary.

Read how How Large Language Models Work explains token →

Build Python Web Apps with Streamlit

A token is a chunk of text, such as a word or word fragment, processed as an input or generated as an output by a language model.

How it is used
Uses tokens to explain response generation, API costs, and context-window limits in AI applications.
How deep it goes
Defines tokens and explains next-token probabilities and practical limits, without detailing how text is split into tokens.
Examples
Uses next-token probability options, a trivia game’s batched requests, and a chatbot response limit as examples.
What you can do after
Operate/configure token usage by counting text, shortening prompts, and setting a response limit with ChatOpenAI.

Read how Build Python Web Apps with Streamlit explains token →

Deep Learning with R, Third Edition

A token is a discrete unit of text processed by a language model, such as a word, subword, character, or punctuation symbol.

How it is used
Provides units for text classification preprocessing and for language-model prediction and generation.
How deep it goes
Builds character, word, and byte-pair tokenizers, including vocabulary construction, indexing, and unknown-token handling.
Examples
Compares character, word, and subword tokenization using Moby Dick passages and a small fox sentence dataset.
What you can do after
Build character, word, and byte-pair tokenizers from scratch in R and inspect vocabulary indices and unknown tokens.

Read how Deep Learning with R, Third Edition explains token →

Grokking AI Applications

A token is a discrete text unit that a language model converts into numerical representations and processes as input or output.

How it is used
Represents text as discrete units that a language model converts into numbers and processes to generate output.
How deep it goes
Defines token types and illustrates embeddings with a tiny vocabulary before comparing GPT-3 scale, without requiring expertise in every architecture detail.
Examples
Uses an eight-token vocabulary and the prompt “The capital of Italy is . . .” to illustrate representation and completion.
What you can do after
Understand how language models represent text as tokens and embeddings and generate text one token at a time.

Read how Grokking AI Applications explains token →

Build a Text-to-Image Generator (from Scratch)

A token is a discrete unit produced by segmenting text, such as a word, punctuation mark, or subword.

How it is used
Uses token sequences to prepare language for transformers and to represent image patches or discrete image codes.
How deep it goes
Works through tokenization, index dictionaries, special tokens, padding, masks, and embeddings; uses pretrained tokenizers for later caption and prompt examples.
Examples
Uses German–English phrases, padded dog and eagle captions, image patches, and lion and tiger prompts.
What you can do after
Build it from scratch for indexed translation inputs, then apply pretrained tokenizers to captions and image-generation prompts.

Read how Build a Text-to-Image Generator (from Scratch) explains token →

AI for Everyday IT

A token is a unit of text or other input processed and billed by a language-model service.

How it is used
Explains tokens as a practical measure of conversation capacity and AI service usage.
How deep it goes
Defines tokens, gives an English size estimate, and explains context limits without requiring mastery of token mechanics.
Examples
Uses an interactive tokenizer and a forgotten favorite-color detail to illustrate token counts and context loss.
What you can do after
Apply token estimates and a tokenizer to plan prompt length, and recognize when a long conversation may lose context.

Read how AI for Everyday IT explains token →

Natural Language Processing in Action, Second Edition

A discrete lexical unit obtained by splitting source text into identifiable pieces, including individual characters in character-based processing.

How it is used
Turns text into countable units for numerical representations used in search, retrieval, classification, and comparison.
How deep it goes
Defines tokens and their role in numerical representations, then notes that simple vocabularies lose sequence and word-form relationships.
Examples
Compares programming-language and natural-language tokens, uses a coin sorter analogy, and names n-grams, punctuation, emoticons, and Chinese characters.
What you can do after
Apply token counts to represent documents numerically for keyword detection, search, retrieval, classification, or comparison.

Read how Natural Language Processing in Action, Second Edition explains token →

Sutskever's List

A discrete unit of text processed by a language model, which may represent a word or a word fragment.

How it is used
Serves as the unit Transformers predict and the unit used to measure training-data volume.
How deep it goes
Defines whole-word and fragment tokens and follows them into numeric representation and prediction, but does not specify a tokenization algorithm.
Examples
Illustrates numeric token IDs and compares GPT-3’s 1.7 training tokens per parameter with Chinchilla’s roughly 20:1 estimate.
What you can do after
Evaluate/understand how token counts shape next-token training and comparisons between model size and training data.

Read how Sutskever's List explains token →

Software Security for Developers

A token is a security artifact carrying authorization or identity information between participants in an authentication or authorization protocol.

How it is used
Supports authentication, authorization, single sign-on, passwordless login, and service-to-service identity across application boundaries.
How deep it goes
Explains token roles, JWT structure, validation, expiration, storage, and replay prevention without developing protocol or framework internals.
Examples
Uses access cards, JWTs, magic links, one-time tokens, and service-to-service requests as concrete token examples.
What you can do after
Evaluate/understand which token role applies and recognize expiration, validation, replay, forgery, phishing, and interception risks.

Read how Software Security for Developers explains token →

Generative AI in Action

A token is a discrete unit into which text or other model input is encoded for processing by a language model.

How it is used
Frames tokens as the unit of LLM input and output and a budget for enterprise cost and performance.
How deep it goes
Explains tokenization and iterative generation, then demonstrates counting and operational management without deep mathematical derivation.
Examples
Uses a sentence about a dog for tokenizer counting and a completion response with prompt, completion, and total token counts.
What you can do after
Apply token counting with tiktoken and manage prompt and response budgets in LLM applications.

Read how Generative AI in Action explains token →

Deep Learning with Python, Third Edition

A discrete unit in a text sequence, such as a word or subword, that serves as the basic item attended over by the model.

How it is used
Uses tokens as units for numeric text preprocessing and as sequence elements in self-attention.
How deep it goes
Defines tokens, demonstrates splitting and indexing, and explains how self-attention builds contextual representations.
Examples
Uses a fox sentence for splitting and indexing and a train-station sentence for contextual attention.
What you can do after
Apply Python’s regex package to split text into tokens and index them with a vocabulary.

Read how Deep Learning with Python, Third Edition explains token →

Learn Generative AI with PyTorch

A discrete text unit produced by tokenization, such as a word, punctuation mark, or special character, used as an element of a sequence.

How it is used
Prepares text and music as indexed sequences for neural-network training and generation.
How deep it goes
Works through cleaning, vocabulary construction, and index mapping, while introducing subword tokenization as a later alternative.
Examples
Uses a cleaned novel, English-to-French phrases, and tokenized piano events as sequence examples.
What you can do after
Build token-to-index dictionaries from scratch and prepare indexed sequences for model training.

Read how Learn Generative AI with PyTorch explains token →

Machines that Think

A discrete unit of text used in computational language processing, such as a character, word fragment, or word.

How it is used
Frames tokens as the units through which language models receive text and generate continuations.
How deep it goes
Explains word, character, and subword trade-offs and outlines byte-pair encoding through repeated frequency-based merges.
Examples
Illustrates subword segmentation with “playfulness” and tests character-level brittleness with letter counting in “strawberry.”
What you can do after
Evaluate how token granularity affects vocabulary size, sequence length, and the distinctions available to a model.

Read how Machines that Think explains token →

Grokking AI Algorithms, Second Edition

A token is a discrete unit of text, such as a word, subword, or symbol, used as an input element for language-model processing.

How it is used
Uses tokens as units for preparing text, predicting language, and training a language model.
How deep it goes
Explains token forms, character-based pair merging, numerical encoding, and context-window batching with worked examples.
Examples
Uses “unbelievable” as a subword example and maps a training sequence between text tokens and numerical IDs.
What you can do after
Understand how tokenization turns text into model-ready units and why tokens need not be whole words.

Read how Grokking AI Algorithms, Second Edition explains token →

Vibe Engineering

A unit of model input text counted when measuring the size and cost of a language-model request.

How it is used
Uses token counts to identify request costs and guide context and output decisions.
How deep it goes
Explains input–output pricing and practical reductions with measured examples, without detailing tokenization.
Examples
Compares interaction-mode token counts and reports a rules-file audit and a 316-query context benchmark.
What you can do after
Evaluate and understand token costs when choosing context, output constraints, and interaction modes.

Read how Vibe Engineering explains token →

Applied Reinforcement Learning

A token is a discrete unit of text, such as a word, word fragment, punctuation mark, or symbol, processed by a language model.

How it is used
Represents the discrete choices a language model generates while producing a response, one token at a time.
How deep it goes
Defines token forms, explains vocabulary IDs, and connects token selection to sequential language generation.
Examples
Uses “Take aspirin today” to illustrate a possible token split and describes generating a response one token at a time.
What you can do after
Understand how a language model converts text into token IDs and selects each next-token action.

Read how Applied Reinforcement Learning explains token →

Time Series Forecasting Using Foundation Models

A token is a discrete representation of a sequence segment, including a fixed-length set of past time steps and associated features.

How it is used
Uses tokens to convert time-series inputs into representations that transformer encoders or decoders can process.
How deep it goes
Explains token construction, embeddings, positional encoding, lagged features, scaling, quantization, and patch-based decoding.
Examples
Uses hourly lagged histories, calendar features, time-series patches, mean-scaled values, and quantization bins as concrete token examples.
What you can do after
Evaluate tokenization choices by considering feature richness, quantization precision, context size, and inference cost.

Read how Time Series Forecasting Using Foundation Models explains token →

The AI Pocket Book

A discrete text unit fed to a language model as input, which may correspond to a subword or word rather than an individual letter.

How it is used
Explains how language models represent input, generate output, and calculate usage costs.
How deep it goes
Defines vocabulary items and traces tokenization, integer IDs, prediction, and stopping, while deferring detailed vocabulary construction.
Examples
Uses dog prompts, hungryish, English and French constitutional text, Telugu, API bills, and end-of-text markers.
What you can do after
Evaluate tokenized prompts, estimate token-related costs, and understand why models generate text piece by piece.

Read how The AI Pocket Book explains token →

MongoDB 8.0 in Action, Third Edition

A token is a text unit produced during analysis for indexing and searching, such as a word, phrase, or symbol.

How it is used
Uses tokens to connect document and query text to searchable terms in Lucene and Atlas Search indexes.
How deep it goes
Defines tokenization and explains analyzers, tokenizers, and filters, but stops short of implementing a tokenizer.
Examples
Illustrates matching with two inspection documents and compares standard, simple, whitespace, language, and keyword analyzers.
What you can do after
Operate/configure it by selecting Atlas Search analyzers for indexed fields and query text.

Read how MongoDB 8.0 in Action, Third Edition explains token →

Build a Large Language Model (From Scratch)

A discrete unit of text that a language model processes, roughly corresponding to words or punctuation marks in this section.

How it is used
Uses tokens as the first preprocessing units in preparing text for decoder-only language-model training.
How deep it goes
Explains word, punctuation, subword, and character units, then connects them to IDs and embeddings without deriving tokenizer internals.
Examples
Demonstrates regular-expression splitting with punctuation, double hyphens, and a 20,479-character short story.
What you can do after
Apply Python regular expressions to split text and prepare token sequences for later ID conversion and embedding lookup.

Read how Build a Large Language Model (From Scratch) explains token →

Starting Data Analytics with Generative AI and Python

A token is a unit of text processed by a language model, such as a word, word fragment, or punctuation mark.

How it is used
Frames tokens as the units needed to understand model input limits and why lengthy conversations or code may lose context.
How deep it goes
Explains token units, subword splitting, model-specific schemes, and context truncation, while deferring token-count estimation.
Examples
Uses “don’t” splitting into “do” and “n’t,” and compares lost context to overhearing only the end of a conversation.
What you can do after
Understand how text becomes model-specific tokens and why exceeding a context window can remove crucial information.

Read how Starting Data Analytics with Generative AI and Python explains token →

Local Image Generation

A token is a text-processing unit used by an AI model that may represent a whole word or a smaller meaningful part of one.

How it is used
Explains why Stable Diffusion prompt wording and spacing affect token counts and how text enters image generation.
How deep it goes
Defines characters, words, tokens, and tokenization, then stops before detailing the complete tokenizer or encoder implementation.
Examples
Uses “sandworm,” “sand worm,” “racetrack,” and “race track” to show character, word, spacing, and token differences.
What you can do after
Evaluate prompt token usage with a token counter and recognize why spacing or word structure changes tokenization.

Read how Local Image Generation explains token →

Build AI-Enhanced Web Apps

A token is a discrete unit of text, such as a word, subword, or character, processed by a language model.

How it is used
Converts natural-language prompts into model-consumable sequences and frames token counts as an input to usage-cost management.
How deep it goes
Defines tokenization as splitting text into tokens and numerical IDs, then situates those IDs in model processing and output conversion.
Examples
Counts tokens in a translation prompt and system message with JavaScript and @dqbd/tiktoken.
What you can do after
Apply token counting with @dqbd/tiktoken to estimate prompt and system-message totals.

Read how Build AI-Enhanced Web Apps explains token →

Data Analysis with LLMs

A token is the atomic unit of text representation processed by a language model, typically corresponding to about four characters.

How it is used
Estimates language-model processing fees and supports model-cost comparisons.
How deep it goes
Defines tokens as atomic text units and gives an approximate character count, without detailing tokenization.
Examples
Estimates annual classification costs for 10,000 daily reviews averaging 100 tokens each.
What you can do after
Evaluate invocation costs by counting tokens used on sample classifications.

Read how Data Analysis with LLMs explains token →

Claude at Work

A token is a unit of text processed by a language model, typically representing part of a word, a word, or punctuation.

How it is used
Tracks the text Claude receives and generates, and helps explain usage and conversation limits.
How deep it goes
Defines tokens as a usage measure, estimates their size, and explains limits without describing tokenization internals.
Examples
Estimates a 1,000-word document at about 1,300 tokens and notes that web or file research can add thousands per turn.
What you can do after
Understand how prompts, responses, and tool use add to token consumption and conversation context.

Read how Claude at Work explains token →

Learn Rust in a Month of Lunches

A token is a lexical unit of source input, such as a number, identifier, punctuation mark, or other syntactic element.

How it is used
Matches macro input patterns before Rust checks the generated output as code.
How deep it goes
Explains literal matching and token-tree, expression, and statement captures, without detailing how Rust tokenizes source.
Examples
Uses the number `6`, a rejected `66`, and an unusual multilingual input to show macro matching.
What you can do after
Understand why macro inputs can fail to match before generated code is checked by the compiler.

Read how Learn Rust in a Month of Lunches explains token →

Deep Learning with PyTorch, Second Edition

A text unit produced by tokenization, which may be a character, word, or subword depending on the tokenizer design.

How it is used
Represents the units that form language-model input and output sequences.
How deep it goes
Defines tokens and demonstrates character mappings, then outlines a tokenizer pipeline without detailing its internals.
Examples
Uses lowercase characters, the name “ada,” a boundary marker, and the words “dog” and “god.”
What you can do after
Apply a word-level tokenizer to train on text, obtain a vocabulary size, and encode or decode sequences.

Read how Deep Learning with PyTorch, Second Edition explains token →

Hugging Face in Action

A token is a unit of text that a machine learning model processes as a single input element.

How it is used
Defines the input units used to preprocess language for NLP models, transformers, and large language models.
How deep it goes
Explains word, subword, and character strategies, then demonstrates BERT token output without deriving tokenizer internals.
Examples
Uses “I love cats,” “unhappiness,” “Hello,” and a BERT tokenizer listing to illustrate token boundaries.
What you can do after
Apply it via Hugging Face Transformers to tokenize text and inspect the resulting model input units.

Read how Hugging Face in Action explains token →

AI Agents in Action, Second Edition

A token is a discrete unit of text represented numerically for processing by a language model.

How it is used
Frames token counts as a practical control for language-model cost, context limits, and output quality.
How deep it goes
Defines text-to-index tokenization and connects next-token prediction to sampling.
Examples
Compares JSON at 13 tokens with equivalent regular text at six tokens.
What you can do after
Apply tiktoken or SDK telemetry to measure token counts and manage budgets and context windows.

Read how AI Agents in Action, Second Edition explains token →

Introduction to Generative AI, Second Edition

A token is a word or subword unit into which a text sequence is divided for computational processing.

How it is used
Encodes text for neural-network processing and can mark API calls that trigger tools.
How deep it goes
Defines tokens as words or partial words and describes special-token use in tool training, without detailing tokenization internals.
Examples
Uses “The cat wore red socks” for translation and special tokens to mark API calls.

Read how Introduction to Generative AI, Second Edition explains token →

Kotlin in Action, 2E

A token is a lexical unit representing either a syntactic symbol or a value in an input language.

How it is used
Supports JSON deserialization by dividing input characters into units that a parser can interpret.
How deep it goes
Defines character and value tokens and places them in the lexer-parser pipeline, without detailing token data structures.
Examples
Uses braces, commas, colons, strings, numbers, Boolean values, and null constants as JSON token examples.
What you can do after
Evaluate how JSON input becomes structured elements before deserialization into Kotlin objects.

Read how Kotlin in Action, 2E explains token →

Just Use Postgres!

A token is a smaller text component produced by parsing a document, such as a word, number, or phrase.

How it is used
Supports PostgreSQL full-text search by supplying parsed components for normalization, storage, indexing, and querying.
How deep it goes
Explains parsing, token types, and normalization outcomes, but stops short of implementing a custom parser.
Examples
Uses the sentence “5 explorers are traveling to a distant galaxy” with ts_debug output and English and Russian configurations.
What you can do after
Diagnose tokenization and normalization with PostgreSQL’s ts_debug function and inspect the resulting parser classifications.

Read how Just Use Postgres! explains token →

LLMs in Production

A token is a discrete unit of text representation processed by a language model.

How it is used
Prepares raw text for language-model training by splitting it into numerical input units.
How deep it goes
Places tokenization in preprocessing and follows the resulting units into model training.
Examples
Gives possible unit sizes from single characters to words, without a worked tokenization example.
What you can do after
Understand how tokenization converts raw text into numerical input for language-model training.

Read how LLMs in Production explains token →

Go in Action, Second Edition

A token is a piece of input data produced by a scanner according to its splitting rules.

How it is used
Represents the units Scanner yields while reading input, with boundaries controlled by a split function.
How deep it goes
Defines tokens through Scanner splitting behavior, including newline defaults and alternatives, but does not detail custom split-function implementation.
Examples
Counts whitespace-separated words by applying strings.Fields to each scanned line.
What you can do after
Apply tokens through bufio.Scanner to read input sequentially and process each resulting line.

Read how Go in Action, Second Edition explains token →

LLM Evaluation and Alignment, The Foundational Ideas

A token is a basic unit of text processed by a language system, such as a word, punctuation mark, or subword piece.

How it is used
Defines the text units counted when BLEU and ROUGE compare generated candidates with references.
How deep it goes
Distinguishes word-based BLEU tokens from modern subword pieces but does not detail tokenization algorithms.
Examples
Illustrates subword splitting with “unhappiness” and consecutive tokens with bigrams such as “the cat.”
What you can do after
Understand how text units become word-overlap counts in BLEU-style evaluation.

Read how LLM Evaluation and Alignment, The Foundational Ideas explains token →

Inside AI

A token is a sequence of characters, word fragment, or related text unit processed by a language model.

How it is used
Supports language-model processing of prompts and prediction of the next word or token.
How deep it goes
Defines tokens as whole words or fragments and gives a rough character rule of thumb.
Examples
Illustrates prediction with an electric-vehicle prompt and possible continuations.
What you can do after
Understand how language models divide text into tokens and predict continuations from preceding text.

Read how Inside AI explains token →

Causal AI

A token is a sequence of characters treated as one unit of input or output by a language model.

How it is used
Represents text as manageable input units and as the units whose sequence probabilities autoregressive language models model.
How deep it goes
Defines tokens, demonstrates GPT-2 tokenization and integer encoding, and raises—but does not resolve—whether attention can learn causal representations.
Examples
Uses “Can LLMs reason counterfactually?” to show punctuation, space markers, subword splits, and token IDs.
What you can do after
Understand how text is split and indexed for model input, and distinguish token-level structure from concept-level causal relations.

Read how Causal AI explains token →

Build a Reasoning Model (From Scratch)

A discrete text unit used as the basic element of language-model generation.

How it is used
Supports text processing, generation, and pretraining by representing language as units for next-token prediction.
How deep it goes
Defines token representation and numerical IDs, while deferring tokenizer mechanics to later treatment.
Examples
Uses the sentence “An LLM can be useful.” and its token sequence to demonstrate subword, word, and punctuation units.
What you can do after
Evaluate/understand how text becomes model input and how pretraining uses token sequences for next-token prediction.

Read how Build a Reasoning Model (From Scratch) explains token →

Designing AI Systems

A token is a unit into which text is divided for processing and usage measurement by an AI model.

How it is used
Tracks model-request consumption and supports token-based cost calculations.
How deep it goes
Distinguishes prompt, completion, total, and cached-token metrics, but does not explain tokenization mechanics.
Examples
Uses request-metric fields for prompt, completion, total, and cached tokens rather than numerical examples.
What you can do after
Evaluate model-request consumption and relate token counts to provider costs.

Read how Designing AI Systems explains token →

Timeless Algorithms, The Foundational Ideas

A next token is the linguistic unit that follows the current context in a sequence being modeled.

How it is used
Frames next-token prediction as a problem of reducing uncertainty with available information.
How deep it goes
Explains how context and learned patterns yield a probability distribution, while naming constraints on prediction quality.
What you can do after
Understand how a neural language model uses context to assign probabilities to possible continuations.

Read how Timeless Algorithms, The Foundational Ideas explains token →

Also covered in

What to know first

Related concepts

Questions about token

What is token?

A token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.

Which Manning books explain token?

How Large Language Models Work, Build Python Web Apps with Streamlit, Deep Learning with R, Third Edition, Grokking AI Applications, Build a Text-to-Image Generator (from Scratch), AI for Everyday IT, Natural Language Processing in Action, Second Edition, Sutskever's List and 32 more.

What should I know before learning token?

Manning books that teach token usually introduce language model, prompt, GPT model and transformer architecture first.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.