Vocabulary
A vocabulary is the finite set of distinct tokens a language-processing system can represent, often associated with identifiers.
Natural Language Processing in Action, Second Edition
A lexicon is a collection or index of words and other lexical sequences known to a language-processing system.
- How it is used
- Uses vocabulary as the shared index for token counting, document vectors, and search.
- How deep it goes
- Explains token choices, corpus-wide vocabulary construction, vector dimensions, and information lost through exclusion or separate word forms.
- Examples
- Builds token counts for paragraphs about algorithmic bias and inspects a corpus vocabulary of 246 entries.
- What you can do after
- Apply vocabulary construction using spaCy tokenization, Python Counter, and pandas document-count vectors.
Read how Natural Language Processing in Action, Second Edition explains vocabulary →
Deep Learning with R, Third Edition
A vocabulary is the finite set of tokens that a tokenizer recognizes and assigns indices for text representation.
- How it is used
- Maps text tokens to indices for classification inputs and defines possible next-token outputs for language models.
- How deep it goes
- Builds frequency-limited word and character vocabularies and a byte-pair vocabulary, then applies learned and pretrained vocabularies.
- Examples
- Uses Moby Dick token counts and IMDb review classification to compare vocabulary choices.
- What you can do after
- Builds vocabularies from scratch and applies learned vocabularies with Keras TextVectorization.
Read how Deep Learning with R, Third Edition explains vocabulary →
Learn Generative AI with PyTorch
The number of distinct tokens available in a model's vocabulary.
- How it is used
- Uses vocabularies to turn text and music into indexed sequences for generation and translation models.
- How deep it goes
- Works through cleaning, counting and bidirectional dictionaries, while deferring the details of music-event conversion.
- Examples
- Uses Anna Karenina, English-to-French phrases and MIDI music events to illustrate distinct vocabularies.
- What you can do after
- Builds vocabulary dictionaries from token counts and applies them to encode and recover token sequences.
Read how Learn Generative AI with PyTorch explains vocabulary →
Starting Data Analytics with Generative AI and Python
A vocabulary is the fixed collection of tokens that a language model can represent.
- How it is used
- Uses lexicons to supplement keywords and identify sentiment or other patterns in text.
- How deep it goes
- Explains lexicon selection, file loading, scoring assumptions, and preprocessing adjustments without detailing language-model vocabulary internals.
- Examples
- Uses Portuguese product reviews and SentiLex files to illustrate format mismatches and lemma-based matching.
- What you can do after
- Evaluate and apply lexicons by checking their suitability, adapting loading code, and aligning review preprocessing with term forms.
Read how Starting Data Analytics with Generative AI and Python explains vocabulary →
Inside Deep Learning: Math, Algorithms, Models
The set of distinct characters in the text corpus that is mapped to discrete symbols for the character-level model.
- How it is used
- Maps corpus characters to integer IDs for embedding inputs and next-character targets in an autoregressive model.
- How deep it goes
- Works through dictionary construction, reverse lookup, vocabulary size, and preprocessing choices; leaves broader embedding applications aside.
- Examples
- Uses a 36-character Shakespeare vocabulary and a word-level English–French vocabulary with padding and sentence-boundary tokens.
- What you can do after
- Build it from scratch with Python dictionaries, then use its IDs to prepare PyTorch datasets and size an embedding layer.
Read how Inside Deep Learning: Math, Algorithms, Models explains vocabulary →
How Large Language Models Work
A language-model vocabulary is the collection of unique tokens and their identifiers observed or defined for the model.
- How it is used
- Frames vocabulary as the representational boundary governing what language models can process effectively.
- How deep it goes
- Explains construction through normalization, segmentation, identifiers, BPE, resource trade-offs, and representation failures.
- Examples
- Uses space splitting, punctuation, schoolhouse, thoughtful, frequent character pairs, and GPT-3’s 50,257-token vocabulary.
- What you can do after
- Evaluate whether vocabulary coverage, tokenization behavior, and resource costs fit a language-model application.
Read how How Large Language Models Work explains vocabulary →
Grokking AI Algorithms, Second Edition
A vocabulary is a collection of distinct tokens recognized by a language-processing system, often associated with numerical identifiers.
- How it is used
- Uses a vocabulary to encode training text as token IDs and define the choices for next-token prediction.
- How deep it goes
- Walks through character-based byte-pair merges, token IDs, embedding dimensions, and vocabulary-wide prediction scores.
- Examples
- Traces merging i and n in training text and lists a 34-token vocabulary with numerical IDs.
- What you can do after
- Evaluate and understand how token choices and vocabulary size affect text encoding and next-token prediction.
Read how Grokking AI Algorithms, Second Edition explains vocabulary →
Deep Learning with Python, Third Edition
The complete set of tokens considered by a text representation scheme, defining the positions in the output vector.
- How it is used
- Uses vocabularies to index text for classification and define possible next-token outputs for language modeling.
- How deep it goes
- Works through frequency-based vocabulary construction, unknown-token handling, size trade-offs, and Keras preprocessing.
- Examples
- Compares character and word vocabularies on Moby Dick and learns a character vocabulary from Shakespeare text.
- What you can do after
- Build vocabulary mappings from scratch or apply them using Keras TextVectorization.
Read how Deep Learning with Python, Third Edition explains vocabulary →
Build AI Drug Discovery Pipelines
A vocabulary is a mapping between the permitted text tokens and their integer identifiers.
- How it is used
- Converts SMILES strings into integer sequences suitable for molecular autoencoders and other neural networks.
- How deep it goes
- Explains character mappings, substitutions, special tokens, encoding, decoding, padding, and BPE-style vocabulary construction.
- Examples
- Uses CCO, aromatic benzene SMILES, halogen substitutions, special tokens, and frequent-pair merges as concrete examples.
- What you can do after
- Build it from scratch: encode, pad, batch, and decode SMILES sequences with character-level vocabulary logic.
Read how Build AI Drug Discovery Pipelines explains vocabulary →
LLMs in Production
A vocabulary is the set of tokens and their associated identifiers recognized by a tokenizer or language model.
- How it is used
- Guides tokenizer design for custom datasets, model training, and inference compatibility.
- How deep it goes
- Explains vocabulary parameters and trade-offs, but defers the internal algorithms behind BPE, WordPiece, Unigram, and SentencePiece.
- Examples
- Demonstrates custom tokenizer training, encoded identifier sequences, padding-token insertion, and digit-level arithmetic tokenization.
- What you can do after
- Operate/configure Hugging Face and tokenizers implementations by selecting vocabulary parameters, adding special tokens, and encoding batches.
Applied Reinforcement Learning
A language model’s vocabulary is the set of tokens it can represent and select as output.
- How it is used
- Treats the vocabulary as the discrete action set from which a language model selects each next token.
- How deep it goes
- Traces tokenization into IDs and repeated next-token selection, then maps the model’s output to a categorical policy.
- Examples
- Uses “Take aspirin today” to illustrate tokenization and a lisinopril query to illustrate next-token choices.
- What you can do after
- Understand how vocabulary choices produce text through sequential next-token generation.
Read how Applied Reinforcement Learning explains vocabulary →
The AI Pocket Book
The set of tokens a language model can recognize and use for tokenization and generation.
- How it is used
- Explains how language models convert prompts into valid units for reading, generation, probability calculation, and token-based billing.
- How deep it goes
- Defines token categories and vocabulary construction, then illustrates tokenization costs without detailing the full byte-pair encoding procedure.
- Examples
- Uses English and French Constitution preambles, unfamiliar word formation, non-Latin characters, and a PyTorch embedding example.
- What you can do after
- Evaluate/understand how vocabulary choices affect token counts, language coverage, model cost, and contextual interpretation.
Build a Large Language Model (From Scratch)
A mapping between unique text tokens and unique integer identifiers used to encode and decode text.
- How it is used
- Supports text encoding before embedding and text decoding after language-model generation.
- How deep it goes
- Defines construction and inverse lookup, then implements encoding, decoding, unknown handling, and document separators without descending into embeddings.
- Examples
- Uses sorted Python dictionaries, Edith Wharton text, SimpleTokenizerV1 and V2, unknown words, and concatenated independent texts.
- What you can do after
- Apply it via Python to build vocabularies and encode or decode text with unknown and document-boundary tokens.
Read how Build a Large Language Model (From Scratch) explains vocabulary →
Deep Learning with PyTorch, Second Edition
The set of distinct symbols or words a language model can assign probabilities to in this character- or word-based setting.
- How it is used
- Uses a defined token set to encode text and determine the classes available for language-model generation.
- How deep it goes
- Explains token mappings and tensor construction, then stops short of efficient embeddings and intermediate subword representations.
- Examples
- Demonstrates lowercase character names, a boundary token, cleaned words, dictionary indices, and an 11-by-7,261 one-hot tensor.
- What you can do after
- Build it from scratch with Python dictionaries, encode names or sentences as integer sequences, and construct one-hot tensors.
Read how Deep Learning with PyTorch, Second Edition explains vocabulary →
Machines that Think
A vocabulary is the finite set of tokens that a language-processing model or tokenizer can represent.
- How it is used
- Defines the inventory of token units a tokenizer can use to represent text.
- How deep it goes
- Contrasts word, character, and subword inventories, then explains BPE merges and WordPiece’s alternative criterion.
- Examples
- Uses “low” and “est” as BPE units and segments “playfulness” into “play,” “ful,” and “ness.”
- What you can do after
- Understand how token granularity affects vocabulary size and the representation of unfamiliar words.
Data Without Labels
A vocabulary is the set of distinct words or tokens recognized in a corpus or text-processing system.
- How it is used
- Organizes tokens into a corpus-level representation for bag-of-words text processing.
- How deep it goes
- Defines unique corpus tokens, frequency thresholds, and fixed-length vectors, but does not preserve word order or context.
- Examples
- Uses two short sentences to demonstrate vocabulary frequencies and zero-filled sentence vectors.
- What you can do after
- Build it from scratch by extracting unique tokens and encoding sentences as vocabulary-sized frequency vectors.
Understanding AI
A vocabulary is the finalized collection of tokens that a language model can represent and process.
- How it is used
- Explains how written input becomes the token sequence processed by a language model.
- How deep it goes
- Traces BPE from character tokens through frequent-pair merges and arbitrary IDs, stopping before embeddings.
- Examples
- Uses “the,” “is,” and “unbelievable” to show single-token and subword representations.
- What you can do after
- Understand how token vocabularies are built and why token IDs do not encode meaning.
Also covered in
- LLM Customization and Fine-TuningIn early access (MEAP)
- Time Series Forecasting Using Foundation Models
- Build a Text-to-Image Generator (from Scratch)
- Think Like a CTO
- Learn Haskell by Example
- Causal AI
- Grokking Machine Learning, Second EditionIn early access (MEAP)
- HTML5 in Action
- Tiny Python Projects: Learn coding and testing with puzzles and games
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- embeddingAn embedding is a numerical vector representation of an item in a feature space, encoding selected properties or relationships for computational tasks such as comparison or retrieval.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
Related concepts
- microdataused by · Microdata is an HTML feature for embedding semantic data in a web page using named vocabularies and properties.
- bag-of-words modelused by · A bag-of-words model represents text using word or token occurrences, counts, or weights while disregarding their order and syntactic relationships.
- dictionary-based methodused by · A dictionary-based method analyzes text by matching its terms against a predefined dictionary or lexicon of relevant words and expressions.
- dimensionequivalent to · A dimension is an independent coordinate or feature axis used to represent points or data; the dimensionality of a representation is the number of such axes or components.
- lemmahas part · A lemma is the canonical dictionary form representing a group of morphologically related word forms.
Questions about vocabulary
What is vocabulary?
A vocabulary is the finite set of distinct tokens a language-processing system can represent, often associated with identifiers.
Which Manning books explain vocabulary?
Natural Language Processing in Action, Second Edition, Deep Learning with R, Third Edition, Learn Generative AI with PyTorch, Starting Data Analytics with Generative AI and Python, Inside Deep Learning: Math, Algorithms, Models, How Large Language Models Work, Grokking AI Algorithms, Second Edition, Deep Learning with Python, Third Edition and 9 more.
What should I know before learning vocabulary?
Manning books that teach vocabulary usually introduce language model, embedding, token and natural language processing first.
What kinds of vocabulary are there?
Kinds of vocabulary include OPINLEX.
What are the parts of vocabulary?
Vocabulary includes lemma.
What is vocabulary also called?
Vocabulary is also called lexicon or Vocab.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.
















