liveBook · Manning

Vocabulary

Also called lexicon, Vocab

A vocabulary is the finite set of distinct tokens a language-processing system can represent, often associated with identifiers.

How 17 Manning books explain it

Natural Language Processing in Action, Second Edition

A lexicon is a collection or index of words and other lexical sequences known to a language-processing system.

How it is used
Uses vocabulary as the shared index for token counting, document vectors, and search.
How deep it goes
Explains token choices, corpus-wide vocabulary construction, vector dimensions, and information lost through exclusion or separate word forms.
Examples
Builds token counts for paragraphs about algorithmic bias and inspects a corpus vocabulary of 246 entries.
What you can do after
Apply vocabulary construction using spaCy tokenization, Python Counter, and pandas document-count vectors.

Read how Natural Language Processing in Action, Second Edition explains vocabulary →

Deep Learning with R, Third Edition

A vocabulary is the finite set of tokens that a tokenizer recognizes and assigns indices for text representation.

How it is used
Maps text tokens to indices for classification inputs and defines possible next-token outputs for language models.
How deep it goes
Builds frequency-limited word and character vocabularies and a byte-pair vocabulary, then applies learned and pretrained vocabularies.
Examples
Uses Moby Dick token counts and IMDb review classification to compare vocabulary choices.
What you can do after
Builds vocabularies from scratch and applies learned vocabularies with Keras TextVectorization.

Read how Deep Learning with R, Third Edition explains vocabulary →

Learn Generative AI with PyTorch

The number of distinct tokens available in a model's vocabulary.

How it is used
Uses vocabularies to turn text and music into indexed sequences for generation and translation models.
How deep it goes
Works through cleaning, counting and bidirectional dictionaries, while deferring the details of music-event conversion.
Examples
Uses Anna Karenina, English-to-French phrases and MIDI music events to illustrate distinct vocabularies.
What you can do after
Builds vocabulary dictionaries from token counts and applies them to encode and recover token sequences.

Read how Learn Generative AI with PyTorch explains vocabulary →

Starting Data Analytics with Generative AI and Python

A vocabulary is the fixed collection of tokens that a language model can represent.

How it is used
Uses lexicons to supplement keywords and identify sentiment or other patterns in text.
How deep it goes
Explains lexicon selection, file loading, scoring assumptions, and preprocessing adjustments without detailing language-model vocabulary internals.
Examples
Uses Portuguese product reviews and SentiLex files to illustrate format mismatches and lemma-based matching.
What you can do after
Evaluate and apply lexicons by checking their suitability, adapting loading code, and aligning review preprocessing with term forms.

Read how Starting Data Analytics with Generative AI and Python explains vocabulary →

Inside Deep Learning: Math, Algorithms, Models

The set of distinct characters in the text corpus that is mapped to discrete symbols for the character-level model.

How it is used
Maps corpus characters to integer IDs for embedding inputs and next-character targets in an autoregressive model.
How deep it goes
Works through dictionary construction, reverse lookup, vocabulary size, and preprocessing choices; leaves broader embedding applications aside.
Examples
Uses a 36-character Shakespeare vocabulary and a word-level English–French vocabulary with padding and sentence-boundary tokens.
What you can do after
Build it from scratch with Python dictionaries, then use its IDs to prepare PyTorch datasets and size an embedding layer.

Read how Inside Deep Learning: Math, Algorithms, Models explains vocabulary →

How Large Language Models Work

A language-model vocabulary is the collection of unique tokens and their identifiers observed or defined for the model.

How it is used
Frames vocabulary as the representational boundary governing what language models can process effectively.
How deep it goes
Explains construction through normalization, segmentation, identifiers, BPE, resource trade-offs, and representation failures.
Examples
Uses space splitting, punctuation, schoolhouse, thoughtful, frequent character pairs, and GPT-3’s 50,257-token vocabulary.
What you can do after
Evaluate whether vocabulary coverage, tokenization behavior, and resource costs fit a language-model application.

Read how How Large Language Models Work explains vocabulary →

Grokking AI Algorithms, Second Edition

A vocabulary is a collection of distinct tokens recognized by a language-processing system, often associated with numerical identifiers.

How it is used
Uses a vocabulary to encode training text as token IDs and define the choices for next-token prediction.
How deep it goes
Walks through character-based byte-pair merges, token IDs, embedding dimensions, and vocabulary-wide prediction scores.
Examples
Traces merging i and n in training text and lists a 34-token vocabulary with numerical IDs.
What you can do after
Evaluate and understand how token choices and vocabulary size affect text encoding and next-token prediction.

Read how Grokking AI Algorithms, Second Edition explains vocabulary →

Deep Learning with Python, Third Edition

The complete set of tokens considered by a text representation scheme, defining the positions in the output vector.

How it is used
Uses vocabularies to index text for classification and define possible next-token outputs for language modeling.
How deep it goes
Works through frequency-based vocabulary construction, unknown-token handling, size trade-offs, and Keras preprocessing.
Examples
Compares character and word vocabularies on Moby Dick and learns a character vocabulary from Shakespeare text.
What you can do after
Build vocabulary mappings from scratch or apply them using Keras TextVectorization.

Read how Deep Learning with Python, Third Edition explains vocabulary →

Build AI Drug Discovery Pipelines

A vocabulary is a mapping between the permitted text tokens and their integer identifiers.

How it is used
Converts SMILES strings into integer sequences suitable for molecular autoencoders and other neural networks.
How deep it goes
Explains character mappings, substitutions, special tokens, encoding, decoding, padding, and BPE-style vocabulary construction.
Examples
Uses CCO, aromatic benzene SMILES, halogen substitutions, special tokens, and frequent-pair merges as concrete examples.
What you can do after
Build it from scratch: encode, pad, batch, and decode SMILES sequences with character-level vocabulary logic.

Read how Build AI Drug Discovery Pipelines explains vocabulary →

LLMs in Production

A vocabulary is the set of tokens and their associated identifiers recognized by a tokenizer or language model.

How it is used
Guides tokenizer design for custom datasets, model training, and inference compatibility.
How deep it goes
Explains vocabulary parameters and trade-offs, but defers the internal algorithms behind BPE, WordPiece, Unigram, and SentencePiece.
Examples
Demonstrates custom tokenizer training, encoded identifier sequences, padding-token insertion, and digit-level arithmetic tokenization.
What you can do after
Operate/configure Hugging Face and tokenizers implementations by selecting vocabulary parameters, adding special tokens, and encoding batches.

Read how LLMs in Production explains vocabulary →

Applied Reinforcement Learning

A language model’s vocabulary is the set of tokens it can represent and select as output.

How it is used
Treats the vocabulary as the discrete action set from which a language model selects each next token.
How deep it goes
Traces tokenization into IDs and repeated next-token selection, then maps the model’s output to a categorical policy.
Examples
Uses “Take aspirin today” to illustrate tokenization and a lisinopril query to illustrate next-token choices.
What you can do after
Understand how vocabulary choices produce text through sequential next-token generation.

Read how Applied Reinforcement Learning explains vocabulary →

The AI Pocket Book

The set of tokens a language model can recognize and use for tokenization and generation.

How it is used
Explains how language models convert prompts into valid units for reading, generation, probability calculation, and token-based billing.
How deep it goes
Defines token categories and vocabulary construction, then illustrates tokenization costs without detailing the full byte-pair encoding procedure.
Examples
Uses English and French Constitution preambles, unfamiliar word formation, non-Latin characters, and a PyTorch embedding example.
What you can do after
Evaluate/understand how vocabulary choices affect token counts, language coverage, model cost, and contextual interpretation.

Read how The AI Pocket Book explains vocabulary →

Build a Large Language Model (From Scratch)

A mapping between unique text tokens and unique integer identifiers used to encode and decode text.

How it is used
Supports text encoding before embedding and text decoding after language-model generation.
How deep it goes
Defines construction and inverse lookup, then implements encoding, decoding, unknown handling, and document separators without descending into embeddings.
Examples
Uses sorted Python dictionaries, Edith Wharton text, SimpleTokenizerV1 and V2, unknown words, and concatenated independent texts.
What you can do after
Apply it via Python to build vocabularies and encode or decode text with unknown and document-boundary tokens.

Read how Build a Large Language Model (From Scratch) explains vocabulary →

Deep Learning with PyTorch, Second Edition

The set of distinct symbols or words a language model can assign probabilities to in this character- or word-based setting.

How it is used
Uses a defined token set to encode text and determine the classes available for language-model generation.
How deep it goes
Explains token mappings and tensor construction, then stops short of efficient embeddings and intermediate subword representations.
Examples
Demonstrates lowercase character names, a boundary token, cleaned words, dictionary indices, and an 11-by-7,261 one-hot tensor.
What you can do after
Build it from scratch with Python dictionaries, encode names or sentences as integer sequences, and construct one-hot tensors.

Read how Deep Learning with PyTorch, Second Edition explains vocabulary →

Machines that Think

A vocabulary is the finite set of tokens that a language-processing model or tokenizer can represent.

How it is used
Defines the inventory of token units a tokenizer can use to represent text.
How deep it goes
Contrasts word, character, and subword inventories, then explains BPE merges and WordPiece’s alternative criterion.
Examples
Uses “low” and “est” as BPE units and segments “playfulness” into “play,” “ful,” and “ness.”
What you can do after
Understand how token granularity affects vocabulary size and the representation of unfamiliar words.

Read how Machines that Think explains vocabulary →

Data Without Labels

A vocabulary is the set of distinct words or tokens recognized in a corpus or text-processing system.

How it is used
Organizes tokens into a corpus-level representation for bag-of-words text processing.
How deep it goes
Defines unique corpus tokens, frequency thresholds, and fixed-length vectors, but does not preserve word order or context.
Examples
Uses two short sentences to demonstrate vocabulary frequencies and zero-filled sentence vectors.
What you can do after
Build it from scratch by extracting unique tokens and encoding sentences as vocabulary-sized frequency vectors.

Read how Data Without Labels explains vocabulary →

Understanding AI

A vocabulary is the finalized collection of tokens that a language model can represent and process.

How it is used
Explains how written input becomes the token sequence processed by a language model.
How deep it goes
Traces BPE from character tokens through frequent-pair merges and arbitrary IDs, stopping before embeddings.
Examples
Uses “the,” “is,” and “unbelievable” to show single-token and subword representations.
What you can do after
Understand how token vocabularies are built and why token IDs do not encode meaning.

Read how Understanding AI explains vocabulary →

Also covered in

What to know first

Related concepts

Questions about vocabulary

What is vocabulary?

A vocabulary is the finite set of distinct tokens a language-processing system can represent, often associated with identifiers.

Which Manning books explain vocabulary?

Natural Language Processing in Action, Second Edition, Deep Learning with R, Third Edition, Learn Generative AI with PyTorch, Starting Data Analytics with Generative AI and Python, Inside Deep Learning: Math, Algorithms, Models, How Large Language Models Work, Grokking AI Algorithms, Second Edition, Deep Learning with Python, Third Edition and 9 more.

What should I know before learning vocabulary?

Manning books that teach vocabulary usually introduce language model, embedding, token and natural language processing first.

What kinds of vocabulary are there?

Kinds of vocabulary include OPINLEX.

What are the parts of vocabulary?

Vocabulary includes lemma.

What is vocabulary also called?

Vocabulary is also called lexicon or Vocab.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.