A vocabulary is the set of distinct tokens a model can use; its size is the number of entries, not the number of tokens in the training sequence. Each entry receives an integer index so text or music can be represented as numbers for a neural network. A reverse mapping turns predicted indexes back into tokens. section 8.3.1 section 14.1.3
To build a text vocabulary, make token boundaries consistent before counting:
- Replace line breaks with spaces and lowercase words, so line breaks are excluded and “The” and “the” share an entry. section 8.3.1
- Separate punctuation from adjacent words: otherwise “way.” and “way” become different tokens. section 8.3.1
- Count unique tokens, assign indexes, and create both token-to-index and index-to-token dictionaries. section 8.3.1
Counting tokens versus vocabulary entries
A cleaned novel yields 437,098 tokens but only 12,778 unique tokens. Sorting the unique tokens by frequency assigns the comma index 0, the period index 1, and “the” index 2. The first count measures sequence length; the second determines how many distinct choices the vocabulary contains. section 8.3.1
Listing 12.4 Mapping tokens to indexes
from collections import Counter
word_counts=Counter(text_tokenized)
words=sorted(word_counts, key=word_counts.get,
reverse=True)
words.append("UNK") #1
text_length=len(text_tokenized)
ntokens=len(words) #2
print(f"the text contains {text_length} words")
print(f"there are {ntokens} unique tokens")
word_to_int={v:k for k,v in enumerate(words)} #3
int_to_word={v:k for k,v in word_to_int.items()} #4
print({k:v for k,v in word_to_int.items() if k in words[:10]})
print({k:v for k,v in int_to_word.items() if v in words[:10]})
The mapping code counts tokens, sorts them by frequency, appends UNK for unseen prompt tokens, and constructs dictionaries in both directions. Here, 10,599 observed tokens become a vocabulary of 10,600 entries. section 12.2.1
Tokenization determines what an entry means. A word-level text generator uses words and separated punctuation; an English-to-French translator instead uses subword tokens, so “French” can become “fr” and “ench.” Translation builds separate English and French mappings, with PAD and UNK entries and BOS and EOS sentence markers. section 12.2 section 10.1.1
Vocabulary size also enters model design. A small GPT configuration sets vocab_size to its 10,600-token vocabulary; generation then predicts an index that can be mapped back to text. Music uses the same idea, but its entries are events rather than words: converting MIDI notes into note-on, note-off, time-shift, and velocity events limits the choices to 388. The details of that MIDI conversion are left to a processor module. section 12.3.1 section 14.2.2
