A vocabulary is the distinct set of symbols or words a language model can recognize and use. Each element is a token, and each token receives a unique integer representation. section 9.1
The vocabulary determines the model’s encoding space. Character vocabularies are compact, while word vocabularies can produce extremely wide one-hot vectors and require handling words absent from the dictionary. section 4.5.3
A vocabulary supports several related operations:
- Map each token to an integer with a character-to-index or word-to-index dictionary. section 4.5.3 section 9.1
- Represent a name or sentence as a sequence of token indices. section 9.1
- Provide the output classes whose probabilities can be produced by a language model. section 9.1
Character and word vocabularies
For name generation, the vocabulary "$abcdefghijklmnopqrstuvwxyz" assigns "$" index 0 and letters successive indices; “ada” becomes [0, 1, 4, 1, 0], where "$" marks both boundaries. For word-level encoding, a cleaned sentence is mapped into an 11-by-7,261 tensor, with one column for each dictionary word. section 9.1 section 4.5.3
Listing 9.1 Setting up vocabulary and character mappings
# In[]:
vocab = "$abcdefghijklmnopqrstuvwxyz"
vocab_size = len(vocab)
ch_to_i = {char: i for i, char in enumerate(vocab)}
i_to_ch = {i: char for i, char in enumerate(vocab)}
Inspect the compact character vocabulary and its forward and reverse Python mappings. section 9.1
# In[8]:
word_t = torch.zeros(len(words_in_line), len(word2index_dict))
for i, word in enumerate(words_in_line):
word_index = word2index_dict[word]
word_t[i][word_index] = 1
print('{:2} {:4} {}'.format(i, word_index, word))
print(word_t.shape)
# Out[8]:
0 3394 impossible
1 4305 mr
2 813 bennet
3 3394 impossible
4 7078 when
5 3315 i
6 415 am
7 4436 not
8 239 acquainted
9 7148 with
10 3215 him
torch.Size([11, 7261])
Inspect how a word vocabulary becomes a one-hot tensor with one vector per word. section 4.5.3
