Token
A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
How Large Language Models Work
A token is a discrete symbolic unit that represents a subword through a unique numeric identifier for processing by a language model.
- How it is used
- Uses tokens as the numeric input units through which an LLM represents, learns from, and generates text.
- How deep it goes
- Explains tokenization through normalization, segmentation, BPE, vocabulary trade-offs, and failure cases, while stopping short of implementation code.
- Examples
- Uses schoolhouse, thoughtful, running, misspelled drug names, character puzzles, and digit-level arithmetic tokenization.
- What you can do after
- Evaluate tokenization-related risks in applications involving spelling, characters, mathematics, languages, cost, or domain vocabulary.
Build Python Web Apps with Streamlit
A token is a chunk of text, such as a word or word fragment, processed as an input or generated as an output by a language model.
- How it is used
- Uses tokens to explain response variability, API costs, response limits, and prompt size.
- How deep it goes
- Defines input and output tokens and explains next-token probabilities, but does not describe how text is split into tokens.
- Examples
- Uses a multi-question trivia app, a token-counting tool, and a chatbot response cap.
- What you can do after
- Operate/configure token-related limits and estimate how prompt and reply lengths affect API costs.
Read how Build Python Web Apps with Streamlit explains token →
Deep Learning with R, Third Edition
A token is a discrete unit of text processed by a language model, such as a word, subword, character, or punctuation symbol.
- How it is used
- Turns text into discrete units for numeric preprocessing and next-token prediction.
- How deep it goes
- Builds character, word, and byte-pair tokenizers, including vocabulary construction and integer lookup.
- Examples
- Compares character and word vocabularies on Moby Dick and demonstrates byte-pair merges on short fox sentences.
- What you can do after
- Build it from scratch: split text, construct a bounded vocabulary, and map tokens to indices using R.
Read how Deep Learning with R, Third Edition explains token →
Grokking AI Applications
A token is a discrete text unit that a language model converts into numerical representations and processes as input or output.
- How it is used
- Explains how language becomes numerical input and one-token-at-a-time output for LLM computation.
- How deep it goes
- Defines tokenization and embeddings, illustrates next-token generation, and stops short of detailed tokenizer or transformer internals.
- Examples
- Uses words, punctuation, “The capital of Italy is ...”, an eight-token vocabulary, and GPT-3’s larger vocabulary as examples.
- What you can do after
- Evaluate how prompts become numerical representations and how LLMs build responses through repeated next-token prediction.
Build a Text-to-Image Generator (from Scratch)
A token is a discrete unit produced by segmenting text, such as a word, punctuation mark, or subword.
- How it is used
- Uses token sequences to prepare language for transformers and to represent image patches or discrete image codes.
- How deep it goes
- Works through tokenization, index dictionaries, special tokens, padding, masks, and embeddings; uses pretrained tokenizers for later caption and prompt examples.
- Examples
- Uses German–English phrases, padded dog and eagle captions, image patches, and lion and tiger prompts.
- What you can do after
- Build it from scratch for indexed translation inputs, then apply pretrained tokenizers to captions and image-generation prompts.
Read how Build a Text-to-Image Generator (from Scratch) explains token →
Natural Language Processing in Action, Second Edition
A discrete lexical unit obtained by splitting source text into identifiable pieces, including individual characters in character-based processing.
- How it is used
- Turns text into countable units for numerical representations used in search, retrieval, classification, and comparison.
- How deep it goes
- Defines tokens and their role in numerical representations, then notes that simple vocabularies lose sequence and word-form relationships.
- Examples
- Compares programming-language and natural-language tokens, uses a coin sorter analogy, and names n-grams, punctuation, emoticons, and Chinese characters.
- What you can do after
- Apply token counts to represent documents numerically for keyword detection, search, retrieval, classification, or comparison.
Read how Natural Language Processing in Action, Second Edition explains token →
Sutskever's List
A discrete unit of text processed by a language model, which may represent a word or a word fragment.
- How it is used
- Serves as the unit Transformers predict and the unit used to measure training-data volume.
- How deep it goes
- Defines whole-word and fragment tokens and follows them into numeric representation and prediction, but does not specify a tokenization algorithm.
- Examples
- Illustrates numeric token IDs and compares GPT-3’s 1.7 training tokens per parameter with Chinchilla’s roughly 20:1 estimate.
- What you can do after
- Evaluate/understand how token counts shape next-token training and comparisons between model size and training data.
Software Security for Developers
A token is a security artifact carrying authorization or identity information between participants in an authentication or authorization protocol.
- How it is used
- Supports authentication, authorization, single sign-on, passwordless login, and service-to-service identity across application boundaries.
- How deep it goes
- Explains token roles, JWT structure, validation, expiration, storage, and replay prevention without developing protocol or framework internals.
- Examples
- Uses access cards, JWTs, magic links, one-time tokens, and service-to-service requests as concrete token examples.
- What you can do after
- Evaluate/understand which token role applies and recognize expiration, validation, replay, forgery, phishing, and interception risks.
Generative AI in Action
A token is a discrete unit into which text or other model input is encoded for processing by a language model.
- How it is used
- Treats tokens as the units of LLM input and output and as a basis for managing application cost, capacity, and performance.
- How deep it goes
- Defines tokenization, traces token-by-token generation, and demonstrates counting with tiktoken without deriving tokenizer or model internals.
- Examples
- Counts a short sentence with tiktoken and inspects prompt, completion, and total token counts in an API response.
- What you can do after
- Apply it via tiktoken to estimate prompt size, then use API usage counts and inference metrics to monitor deployments.
Deep Learning with Python, Third Edition
A discrete unit in a text sequence, such as a word or subword, that serves as the basic item attended over by the model.
- How it is used
- Turns raw text into indexed sequence units that can serve as inputs and as positions updated by self-attention.
- How deep it goes
- Works through regex splitting and vocabulary lookup, then explains token-level self-attention; mentions subwords without specifying a subword tokenizer.
- Examples
- Uses a fox-and-dog sentence for splitting and indexing, and a train-station ambiguity for contextual attention.
- What you can do after
- Apply it via Python regex and vocabulary lookups to split and index text, then understand what self-attention operates over.
Read how Deep Learning with Python, Third Edition explains token →
Learn Generative AI with PyTorch
A discrete text unit produced by tokenization, such as a word, punctuation mark, or special character, used as an element of a sequence.
- How it is used
- Converts text or musical events into indexed sequences for numerical generation models.
- How deep it goes
- Works through word-level cleaning and reversible index mapping, while deferring more sophisticated subword tokenizers.
- Examples
- Uses novel punctuation and frequency counts alongside MIDI time-shift, velocity, and note events.
- What you can do after
- Build it from scratch: clean and tokenize text, create reversible token–index dictionaries, and prepare indexed model inputs.
Grokking AI Algorithms, Second Edition
A token is a discrete unit of text, such as a word, subword, or symbol, used as an input element for language-model processing.
- How it is used
- Serves as the text unit converted into IDs for language-model training and prediction.
- How deep it goes
- Works through character initialization, frequent-pair merging, vocabulary IDs, and context-window batching with a learning-oriented tokenizer.
- Examples
- Uses word-level predictions from five sentences and a Python tokenizer that merges frequently adjacent character tokens.
- What you can do after
- Build it from scratch: form a simple token vocabulary, assign IDs, and divide an ID stream into context-sized inputs.
Read how Grokking AI Algorithms, Second Edition explains token →
Time Series Forecasting Using Foundation Models
A token is a discrete representation of a sequence segment, including a fixed-length set of past time steps and associated features.
- How it is used
- Uses tokens to convert time-series inputs into representations that transformer encoders or decoders can process.
- How deep it goes
- Explains token construction, embeddings, positional encoding, lagged features, scaling, quantization, and patch-based decoding.
- Examples
- Uses hourly lagged histories, calendar features, time-series patches, mean-scaled values, and quantization bins as concrete token examples.
- What you can do after
- Evaluate tokenization choices by considering feature richness, quantization precision, context size, and inference cost.
Read how Time Series Forecasting Using Foundation Models explains token →
The AI Pocket Book
A discrete text unit fed to a language model as input, which may correspond to a subword or word rather than an individual letter.
- How it is used
- Explains how language models represent input, generate output, and calculate usage costs.
- How deep it goes
- Defines vocabulary items and traces tokenization, integer IDs, prediction, and stopping, while deferring detailed vocabulary construction.
- Examples
- Uses dog prompts, hungryish, English and French constitutional text, Telugu, API bills, and end-of-text markers.
- What you can do after
- Evaluate tokenized prompts, estimate token-related costs, and understand why models generate text piece by piece.
MongoDB 8.0 in Action, Third Edition
A token is a text unit produced during analysis for indexing and searching, such as a word, phrase, or symbol.
- How it is used
- Uses tokens to connect document and query text to searchable terms in Lucene and Atlas Search indexes.
- How deep it goes
- Defines tokenization and explains analyzers, tokenizers, and filters, but stops short of implementing a tokenizer.
- Examples
- Illustrates matching with two inspection documents and compares standard, simple, whitespace, language, and keyword analyzers.
- What you can do after
- Operate/configure it by selecting Atlas Search analyzers for indexed fields and query text.
Read how MongoDB 8.0 in Action, Third Edition explains token →
Build a Large Language Model (From Scratch)
A discrete unit of text that a language model processes, roughly corresponding to words or punctuation marks in this section.
- How it is used
- Uses tokens as the first preprocessing units in preparing text for decoder-only language-model training.
- How deep it goes
- Explains word, punctuation, subword, and character units, then connects them to IDs and embeddings without deriving tokenizer internals.
- Examples
- Demonstrates regular-expression splitting with punctuation, double hyphens, and a 20,479-character short story.
- What you can do after
- Apply Python regular expressions to split text and prepare token sequences for later ID conversion and embedding lookup.
Read how Build a Large Language Model (From Scratch) explains token →
Starting Data Analytics with Generative AI and Python
A token is a unit of text processed by a language model, such as a word, word fragment, or punctuation mark.
- How it is used
- Frames tokens as the units needed to understand model input limits and why lengthy conversations or code may lose context.
- How deep it goes
- Explains token units, subword splitting, model-specific schemes, and context truncation, while deferring token-count estimation.
- Examples
- Uses “don’t” splitting into “do” and “n’t,” and compares lost context to overhearing only the end of a conversation.
- What you can do after
- Understand how text becomes model-specific tokens and why exceeding a context window can remove crucial information.
Read how Starting Data Analytics with Generative AI and Python explains token →
Build AI-Enhanced Web Apps
A token is a discrete unit of text, such as a word, subword, or character, processed by a language model.
- How it is used
- Converts natural-language prompts into model-consumable sequences and frames token counts as an input to usage-cost management.
- How deep it goes
- Defines tokenization as splitting text into tokens and numerical IDs, then situates those IDs in model processing and output conversion.
- Examples
- Counts tokens in a translation prompt and system message with JavaScript and @dqbd/tiktoken.
- What you can do after
- Apply token counting with @dqbd/tiktoken to estimate prompt and system-message totals.
Data Analysis with LLMs
A token is the atomic unit of text representation processed by a language model, typically corresponding to about four characters.
- How it is used
- Estimates language-model processing fees and supports model-cost comparisons.
- How deep it goes
- Defines tokens as atomic text units and gives an approximate character count, without detailing tokenization.
- Examples
- Estimates annual classification costs for 10,000 daily reviews averaging 100 tokens each.
- What you can do after
- Evaluate invocation costs by counting tokens used on sample classifications.
Learn Rust in a Month of Lunches
A token is a lexical unit of source input, such as a number, identifier, punctuation mark, or other syntactic element.
- How it is used
- Matches macro input patterns before Rust checks the generated output as code.
- How deep it goes
- Explains literal matching and token-tree, expression, and statement captures, without detailing how Rust tokenizes source.
- Examples
- Uses the number `6`, a rejected `66`, and an unusual multilingual input to show macro matching.
- What you can do after
- Understand why macro inputs can fail to match before generated code is checked by the compiler.
Deep Learning with PyTorch, Second Edition
A text unit produced by tokenization, which may be a character, word, or subword depending on the tokenizer design.
- How it is used
- Represents the units that form language-model input and output sequences.
- How deep it goes
- Defines tokens and demonstrates character mappings, then outlines a tokenizer pipeline without detailing its internals.
- Examples
- Uses lowercase characters, the name “ada,” a boundary marker, and the words “dog” and “god.”
- What you can do after
- Apply a word-level tokenizer to train on text, obtain a vocabulary size, and encode or decode sequences.
Read how Deep Learning with PyTorch, Second Edition explains token →
Hugging Face in Action
A token is a unit of text that a machine learning model processes as a single input element.
- How it is used
- Defines the input units used to preprocess language for NLP models, transformers, and large language models.
- How deep it goes
- Explains word, subword, and character strategies, then demonstrates BERT token output without deriving tokenizer internals.
- Examples
- Uses “I love cats,” “unhappiness,” “Hello,” and a BERT tokenizer listing to illustrate token boundaries.
- What you can do after
- Apply it via Hugging Face Transformers to tokenize text and inspect the resulting model input units.
AI Agents in Action, Second Edition
A token is a discrete unit of text represented numerically for processing by a language model.
- How it is used
- Frames token counts as a practical control for language-model cost, context limits, and output quality.
- How deep it goes
- Defines text-to-index tokenization and connects next-token prediction to sampling.
- Examples
- Compares JSON at 13 tokens with equivalent regular text at six tokens.
- What you can do after
- Apply tiktoken or SDK telemetry to measure token counts and manage budgets and context windows.
Read how AI Agents in Action, Second Edition explains token →
Introduction to Generative AI, Second Edition
A token is a word or subword unit into which a text sequence is divided for computational processing.
- How it is used
- Encodes text for neural-network processing and can mark API calls that trigger tools.
- How deep it goes
- Defines tokens as words or partial words and describes special-token use in tool training, without detailing tokenization internals.
- Examples
- Uses “The cat wore red socks” for translation and special tokens to mark API calls.
Read how Introduction to Generative AI, Second Edition explains token →
Kotlin in Action, 2E
A token is a lexical unit representing either a syntactic symbol or a value in an input language.
- How it is used
- Supports JSON deserialization by dividing input characters into units that a parser can interpret.
- How deep it goes
- Defines character and value tokens and places them in the lexer-parser pipeline, without detailing token data structures.
- Examples
- Uses braces, commas, colons, strings, numbers, Boolean values, and null constants as JSON token examples.
- What you can do after
- Evaluate how JSON input becomes structured elements before deserialization into Kotlin objects.
Just Use Postgres!
A token is a smaller text component produced by parsing a document, such as a word, number, or phrase.
- How it is used
- Supports PostgreSQL full-text search by supplying parsed components for normalization, storage, indexing, and querying.
- How deep it goes
- Explains parsing, token types, and normalization outcomes, but stops short of implementing a custom parser.
- Examples
- Uses the sentence “5 explorers are traveling to a distant galaxy” with ts_debug output and English and Russian configurations.
- What you can do after
- Diagnose tokenization and normalization with PostgreSQL’s ts_debug function and inspect the resulting parser classifications.
LLMs in Production
A token is a discrete unit of text representation processed by a language model.
- How it is used
- Prepares raw text for language-model training by splitting it into numerical input units.
- How deep it goes
- Places tokenization in preprocessing and follows the resulting units into model training.
- Examples
- Gives possible unit sizes from single characters to words, without a worked tokenization example.
- What you can do after
- Understand how tokenization converts raw text into numerical input for language-model training.
Go in Action, Second Edition
A token is a piece of input data produced by a scanner according to its splitting rules.
- How it is used
- Represents the units Scanner yields while reading input, with boundaries controlled by a split function.
- How deep it goes
- Defines tokens through Scanner splitting behavior, including newline defaults and alternatives, but does not detail custom split-function implementation.
- Examples
- Counts whitespace-separated words by applying strings.Fields to each scanned line.
- What you can do after
- Apply tokens through bufio.Scanner to read input sequentially and process each resulting line.
Inside AI
A token is a sequence of characters, word fragment, or related text unit processed by a language model.
- How it is used
- Supports language-model processing of prompts and prediction of the next word or token.
- How deep it goes
- Defines tokens as whole words or fragments and gives a rough character rule of thumb.
- Examples
- Illustrates prediction with an electric-vehicle prompt and possible continuations.
- What you can do after
- Understand how language models divide text into tokens and predict continuations from preceding text.
Causal AI
A token is a sequence of characters treated as one unit of input or output by a language model.
- How it is used
- Represents text as manageable input units and as the units whose sequence probabilities autoregressive language models model.
- How deep it goes
- Defines tokens, demonstrates GPT-2 tokenization and integer encoding, and raises—but does not resolve—whether attention can learn causal representations.
- Examples
- Uses “Can LLMs reason counterfactually?” to show punctuation, space markers, subword splits, and token IDs.
- What you can do after
- Understand how text is split and indexed for model input, and distinguish token-level structure from concept-level causal relations.
Build a Reasoning Model (From Scratch)
A discrete text unit used as the basic element of language-model generation.
- How it is used
- Supports text processing, generation, and pretraining by representing language as units for next-token prediction.
- How deep it goes
- Defines token representation and numerical IDs, while deferring tokenizer mechanics to later treatment.
- Examples
- Uses the sentence “An LLM can be useful.” and its token sequence to demonstrate subword, word, and punctuation units.
- What you can do after
- Evaluate/understand how text becomes model input and how pretraining uses token sequences for next-token prediction.
Read how Build a Reasoning Model (From Scratch) explains token →
Related concepts
- tokenizationproduced by · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
- language modelproduced by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- context windowmeasured by · A context window is the bounded sequence of tokens or other input information a model can access when processing or generating an output.
- Stable Diffusionused by · Stable Diffusion is a family of latent diffusion models that generates and edits images from text and other conditioning inputs.
- token streamingused by · Token streaming is the incremental delivery of generated language-model output as tokens become available, rather than waiting for the complete output.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.