Tokenization turns raw text into numerical input by splitting it into units that may be as small as a character or as large as a word. The resulting tokenized text is fed into the language model during training. section 5.2.1
Token
A token is a discrete unit of text representation processed by a language model.
How LLMs in Production covers it
- How deep it goes
- Places tokenization in preprocessing and follows the resulting units into model training.
- Examples
- Gives possible unit sizes from single characters to words, without a worked tokenization example.
- What you can do after
- Understand how tokenization converts raw text into numerical input for language-model training.
- How it connects
- Links tokens with preprocessing, numerical representations, and language-model training.
Also explained in
-
How Large Language Models WorkA token is a discrete symbolic unit that represents a subword through a unique numeric identifier for processing by a language model.
-
Build Python Web Apps with StreamlitA token is a chunk of text, such as a word or word fragment, processed as an input or generated as an output by a language model.
-
Deep Learning with R, Third EditionA token is a discrete unit of text processed by a language model, such as a word, subword, character, or punctuation symbol.
-
Grokking AI ApplicationsA token is a discrete text unit that a language model converts into numerical representations and processes as input or output.
-
Build a Text-to-Image Generator (from Scratch)A token is a discrete unit produced by segmenting text, such as a word, punctuation mark, or subword.
-
Natural Language Processing in Action, Second EditionA discrete lexical unit obtained by splitting source text into identifiable pieces, including individual characters in character-based processing.
-
Sutskever's ListA discrete unit of text processed by a language model, which may represent a word or a word fragment.
-
Software Security for DevelopersA token is a security artifact carrying authorization or identity information between participants in an authentication or authorization protocol.
-
Generative AI in ActionA token is a discrete unit into which text or other model input is encoded for processing by a language model.
-
Deep Learning with Python, Third EditionA discrete unit in a text sequence, such as a word or subword, that serves as the basic item attended over by the model.
-
Learn Generative AI with PyTorchA discrete text unit produced by tokenization, such as a word, punctuation mark, or special character, used as an element of a sequence.
-
Grokking AI Algorithms, Second EditionA token is a discrete unit of text, such as a word, subword, or symbol, used as an input element for language-model processing.
-
Time Series Forecasting Using Foundation ModelsA token is a discrete representation of a sequence segment, including a fixed-length set of past time steps and associated features.
-
The AI Pocket BookA discrete text unit fed to a language model as input, which may correspond to a subword or word rather than an individual letter.
-
MongoDB 8.0 in Action, Third EditionA token is a text unit produced during analysis for indexing and searching, such as a word, phrase, or symbol.
-
Build a Large Language Model (From Scratch)A discrete unit of text that a language model processes, roughly corresponding to words or punctuation marks in this section.
-
Starting Data Analytics with Generative AI and PythonA token is a unit of text processed by a language model, such as a word, word fragment, or punctuation mark.
-
Build AI-Enhanced Web AppsA token is a discrete unit of text, such as a word, subword, or character, processed by a language model.
-
Data Analysis with LLMsA token is the atomic unit of text representation processed by a language model, typically corresponding to about four characters.
-
Learn Rust in a Month of LunchesA token is a lexical unit of source input, such as a number, identifier, punctuation mark, or other syntactic element.
-
Deep Learning with PyTorch, Second EditionA text unit produced by tokenization, which may be a character, word, or subword depending on the tokenizer design.
-
Hugging Face in ActionA token is a unit of text that a machine learning model processes as a single input element.
-
AI Agents in Action, Second EditionA token is a discrete unit of text represented numerically for processing by a language model.
-
Introduction to Generative AI, Second EditionA token is a word or subword unit into which a text sequence is divided for computational processing.
-
Kotlin in Action, 2EA token is a lexical unit representing either a syntactic symbol or a value in an input language.
-
Just Use Postgres!A token is a smaller text component produced by parsing a document, such as a word, number, or phrase.
-
Go in Action, Second EditionA token is a piece of input data produced by a scanner according to its splitting rules.
-
Inside AIA token is a sequence of characters, word fragment, or related text unit processed by a language model.
-
Causal AIA token is a sequence of characters treated as one unit of input or output by a language model.
-
Build a Reasoning Model (From Scratch)A discrete text unit used as the basic element of language-model generation.
Compare how 31 books explain token →
Related concepts
- tokenizationproduced by · Tokenization is the process of dividing text or other sequential input into discrete units called tokens for computational processing.
- language modelproduced by · A language model is a computational model that represents patterns in linguistic tokens to estimate sequence probabilities and predict or generate language from context.
- context windowmeasured by · A context window is the bounded sequence of tokens or other input information a model can access when processing or generating an output.
- Stable Diffusionused by · Stable Diffusion is a family of latent diffusion models that generates and edits images from text and other conditioning inputs.
- token streamingused by · Token streaming is the incremental delivery of generated language-model output as tokens become available, rather than waiting for the complete output.
This explainer was generated by Manning's knowledge graph from the text of LLMs in Production. The section links point to the parts of the book it draws on.