Vocabulary
A vocabulary is a collection of distinct words, tokens, or other lexical units recognized or represented by a language or language-processing system.
Natural Language Processing in Action, Second Edition
A lexicon is a collection or index of words and other lexical sequences known to a language-processing system.
- How it is used
- Defines which tokens receive positions in document vectors for counting, search, and other NLP tasks.
- How deep it goes
- Builds a corpus vocabulary from token sets and count tables, then examines normalization, omitted tokens, and vector-size consequences.
- Examples
- Uses spaCy tokens, Python sets and Counters, and a pandas document–token table built from passages about algorithmic bias.
- What you can do after
- Build it from scratch by collecting unique corpus tokens, then use them as consistent dimensions for document count vectors.
Read how Natural Language Processing in Action, Second Edition explains vocabulary →
Deep Learning with R, Third Edition
A vocabulary is the finite set of tokens that a tokenizer recognizes and assigns indices for text representation.
- How it is used
- Maps text tokens to integer indices for classification and next-token prediction.
- How deep it goes
- Works through manual indexing, frequency-limited vocabularies, byte-pair construction, and learned Keras vocabularies.
- Examples
- Uses a fox sentence, Moby Dick token counts, IMDb reviews, and a Shakespeare character vocabulary.
- What you can do after
- Build it from scratch with R token counts and byte-pair merges, or apply it via Keras TextVectorization and SentencePiece.
Read how Deep Learning with R, Third Edition explains vocabulary →
Learn Generative AI with PyTorch
The number of distinct tokens available in a model's vocabulary.
- How it is used
- Maps distinct text or music tokens to integer choices for model inputs and next-token predictions.
- How deep it goes
- Works through cleaning, counting, bidirectional mappings, and special tokens, but defers the details of MIDI event conversion.
- Examples
- Uses novel token counts, English-to-French subwords, and a 388-event MIDI vocabulary.
- What you can do after
- Build it from scratch for text with token-index mappings, then set a model’s vocabulary-size parameter from the resulting count.
Read how Learn Generative AI with PyTorch explains vocabulary →
Starting Data Analytics with Generative AI and Python
A vocabulary is the fixed collection of tokens that a language model can represent.
- How it is used
- Uses a task-specific lexicon to extend keyword searches and support sentiment analysis of reviews.
- How deep it goes
- Works through lexicon selection, file loading, and matching; does not explain language-model token vocabularies.
- Examples
- Uses Portuguese product reviews and SentiLex files to expose a mistaken CSV assumption and a lemma-matching problem.
- What you can do after
- Apply it via a chosen lexicon and Python preprocessing, then inspect file formats and test matches against reviews.
Read how Starting Data Analytics with Generative AI and Python explains vocabulary →
Inside Deep Learning: Math, Algorithms, Models
The set of distinct characters in the text corpus that is mapped to discrete symbols for the character-level model.
- How it is used
- Converts characters or words into discrete IDs so recurrent, autoregressive, and sequence-to-sequence models can consume sequences.
- How deep it goes
- Explains notation, normalization choices, bidirectional mappings, tensor conversion, and special tokens, but defers broader NLP representation methods.
- Examples
- Uses language names, a lowercased Shakespeare corpus, and English-French word pairs with PAD, SOS, and EOS markers.
- What you can do after
- Builds integer encodings and reverse lookups for sequence datasets and configures vocabulary-sized embedding or prediction layers.
Read how Inside Deep Learning: Math, Algorithms, Models explains vocabulary →
How Large Language Models Work
A language-model vocabulary is the collection of unique tokens and their identifiers observed or defined for the model.
- How it is used
- Frames vocabulary as the representational boundary governing what language models can process effectively.
- How deep it goes
- Explains construction through normalization, segmentation, identifiers, BPE, resource trade-offs, and representation failures.
- Examples
- Uses space splitting, punctuation, schoolhouse, thoughtful, frequent character pairs, and GPT-3’s 50,257-token vocabulary.
- What you can do after
- Evaluate whether vocabulary coverage, tokenization behavior, and resource costs fit a language-model application.
Read how How Large Language Models Work explains vocabulary →
Grokking AI Algorithms, Second Edition
A vocabulary is a collection of distinct tokens recognized by a language-processing system, often associated with numerical identifiers.
- How it is used
- Connects text preparation to model training by mapping recognized tokens to numerical IDs.
- How deep it goes
- Works through character tokens and repeated frequent-pair merges, then shows how vocabulary size shapes embeddings and prediction scores.
- Examples
- Uses character-pair counts, merged tokens, a token-to-ID table, and next-token probability tables.
- What you can do after
- Build it from scratch as a small token-to-ID vocabulary using repeated frequent-pair merges.
Read how Grokking AI Algorithms, Second Edition explains vocabulary →
Deep Learning with Python, Third Edition
The complete set of tokens considered by a text representation scheme, defining the positions in the output vector.
- How it is used
- Maps text tokens to numeric IDs and determines representation dimensions for classification and next-token prediction.
- How deep it goes
- Builds capped character and word vocabularies with counting and unknown-token handling, then applies vocabulary sizes to model inputs and outputs.
- Examples
- Compares character and word vocabularies on a novel and learns a 67-character vocabulary for a Shakespeare language model.
- What you can do after
- Build it from scratch with token counts and an unknown-token fallback, or apply it via Keras `TextVectorization`.
Read how Deep Learning with Python, Third Edition explains vocabulary →
LLMs in Production
A vocabulary is the set of tokens and their associated identifiers recognized by a tokenizer or language model.
- How it is used
- Guides tokenizer design for custom datasets, model training, and inference compatibility.
- How deep it goes
- Explains vocabulary parameters and trade-offs, but defers the internal algorithms behind BPE, WordPiece, Unigram, and SentencePiece.
- Examples
- Demonstrates custom tokenizer training, encoded identifier sequences, padding-token insertion, and digit-level arithmetic tokenization.
- What you can do after
- Operate/configure Hugging Face and tokenizers implementations by selecting vocabulary parameters, adding special tokens, and encoding batches.
The AI Pocket Book
The set of tokens a language model can recognize and use for tokenization and generation.
- How it is used
- Explains how language models convert prompts into valid units for reading, generation, probability calculation, and token-based billing.
- How deep it goes
- Defines token categories and vocabulary construction, then illustrates tokenization costs without detailing the full byte-pair encoding procedure.
- Examples
- Uses English and French Constitution preambles, unfamiliar word formation, non-Latin characters, and a PyTorch embedding example.
- What you can do after
- Evaluate/understand how vocabulary choices affect token counts, language coverage, model cost, and contextual interpretation.
Build a Large Language Model (From Scratch)
A mapping between unique text tokens and unique integer identifiers used to encode and decode text.
- How it is used
- Supports text encoding before embedding and text decoding after language-model generation.
- How deep it goes
- Defines construction and inverse lookup, then implements encoding, decoding, unknown handling, and document separators without descending into embeddings.
- Examples
- Uses sorted Python dictionaries, Edith Wharton text, SimpleTokenizerV1 and V2, unknown words, and concatenated independent texts.
- What you can do after
- Apply it via Python to build vocabularies and encode or decode text with unknown and document-boundary tokens.
Read how Build a Large Language Model (From Scratch) explains vocabulary →
Deep Learning with PyTorch, Second Edition
The set of distinct symbols or words a language model can assign probabilities to in this character- or word-based setting.
- How it is used
- Uses a defined token set to encode text and determine the classes available for language-model generation.
- How deep it goes
- Explains token mappings and tensor construction, then stops short of efficient embeddings and intermediate subword representations.
- Examples
- Demonstrates lowercase character names, a boundary token, cleaned words, dictionary indices, and an 11-by-7,261 one-hot tensor.
- What you can do after
- Build it from scratch with Python dictionaries, encode names or sentences as integer sequences, and construct one-hot tensors.
Read how Deep Learning with PyTorch, Second Edition explains vocabulary →
Data Without Labels
A vocabulary is the set of distinct words or tokens recognized in a corpus or text-processing system.
- How it is used
- Organizes tokens into a corpus-level representation for bag-of-words text processing.
- How deep it goes
- Defines unique corpus tokens, frequency thresholds, and fixed-length vectors, but does not preserve word order or context.
- Examples
- Uses two short sentences to demonstrate vocabulary frequencies and zero-filled sentence vectors.
- What you can do after
- Build it from scratch by extracting unique tokens and encoding sentences as vocabulary-sized frequency vectors.
Related concepts
- lemmahas part · A lemma is the canonical dictionary form representing a group of morphologically related word forms.
- dictionary-based methodused by · A dictionary-based method analyzes text by matching its terms against a predefined dictionary or lexicon of relevant words and expressions.
- sentiment analysisuses · Sentiment analysis is the computational identification, classification, or measurement of opinions, attitudes, emotions, or evaluative polarity expressed in text or other data.
- tokenincludes · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
- bag-of-words modelused by · A bag-of-words model represents text using counts, presence indicators, or weights for its words or tokens while disregarding their order and syntactic relationships.
- dimensionequivalent to · A dimension is an independent coordinate, measurable axis, or descriptive attribute used to represent, locate, or analyze something.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.