A token is a basic unit in a model’s input or output sequence. Depending on the tokenizer, it can be a character, word, or subword. A tokenizer splits raw text into tokens and converts them to numbers so a model can process them. section 9.1 section 9.8
Characters as tokens
With lowercase letters as the vocabulary, “ada” becomes [1, 4, 1]. A special $ token can mark both ends, giving [0, 1, 4, 1, 0]. Word-level tokens can carry more meaning than individual characters: “dog” and “god” use the same letters in a different order but have different meanings. section 9.1 section 9.8
Figure 9.1 General terminology in language models relating to vocabulary, token, and sequence. The $ token is a special token that indicates the start or end of a name.
See how vocabulary, tokens, and sequences relate, including the special boundary token. section 9.1
# In[]: from tokenizers import Tokenizer from tokenizers.models import WordLevel from tokenizers.pre_tokenizers import Whitespace from tokenizers.trainers import WordLevelTrainer tokenizer = Tokenizer(WordLevel(unk_token="[UNK]")) #1 tokenizer.pre_tokenizer = Whitespace() #2 trainer = WordLevelTrainer(special_tokens=["[UNK]"]) #3 tokenizer.train(["../data/p2ch9/odyssey.txt"], trainer=trainer) vocab_size = tokenizer.get_vocab_size() decode = tokenizer.decode encode = tokenizer.encode
See a word-level tokenizer setup, including whitespace pre-tokenization and training on text. section 9.8
