Tokenization splits text into discrete tokens such as words, punctuation marks, or subwords. In language models, these tokens become the units used for later numerical encoding. section 2.2.1 section 2.1.1
A practical tokenization pipeline proceeds through these stages:
- Load language-specific tokenizers, such as spaCy's German and English pipelines. section 2.2.1
- Split each phrase into words, punctuation marks, or subwords. section 2.2.1 section 2.1.1
- Add special sequence markers such as BOS and EOS when constructing training-token lists. section 2.2.1
- Count tokens, retain the most frequent vocabulary entries, and reserve PAD and UNK indices. section 2.2.1
- Map tokens to integer IDs before converting them into embeddings or other model inputs. section 2.2.1 section 2.1.1
German-English translation
Tokenizing “Two young, White males are outside near many bushes.” produces tokens including “Two”, “young”, “,” and “bushes”. The tokens can then be mapped to IDs such as [19, 25, 15, 1165, ...]. section 2.2.1
Text-to-image prompts
A prompt such as “panda with top hat reading a book” is tokenized, padded, formatted, and passed to a pretrained encoder to create text embeddings that condition image generation. section 12.2 section 10.3
Listing 2.2 Creating a dictionary to map English tokens to indices
from collections import Counter
en_tokens=[["BOS"]+[tok.text for tok in en_tokenizer.tokenizer(x)] #1
+["EOS"] for x in trainen] #1
PAD=0
UNK=1
word_count=Counter()
for sentence in en_tokens:
for word in sentence:
word_count[word]+=1
frequency=word_count.most_common(50000)
total_en_words=len(frequency)+2
en_word_dict={w[0]:idx+2 for idx,w in enumerate(frequency)} #2
en_word_dict["PAD"]=PAD
en_word_dict["UNK"]=UNK #3
en_idx_dict={v:k for k,v
in en_word_dict.items()} #4
Inspect the vocabulary-building implementation, including frequency counting, special-token indices, and forward and reverse dictionaries. section 2.2.1
