An end-of-text token marks the end of a token sequence. In sentence translation, it is called EOS, for “end of sentence.” It differs from BOS, which marks the beginning, and PAD, which fills shorter sequences to a common length. section 2.1.2 section 2.2.1
To use EOS in a German-to-English transformer:
- Tokenize each training phrase, append EOS, and give it an index in the token dictionary. This lets the model learn where phrases end. section 2.1.2 section 2.2.1
- For translation, place BOS and EOS around the German input. Start the English decoder with BOS, then predict one next token at a time from the tokens generated so far. section 2.4.2
- Stop generating the English translation when the decoder predicts EOS. EOS on the German input marks that input’s end; predicted EOS ends the output. section 2.1.2 section 2.4.2
The same role, a different token spelling
A min-DALL-E text prompt uses </s> as its end-of-sequence token. “panda with top hat reading a book” tokenizes to IDs beginning with 0 (<s>) and ending with 2 (</s>). The prompt can then be padded to length 64 with ID 1 (<pad>). These IDs must match the pretrained model’s vocabulary. section 12.2.1
Listing 2.2 Creating a dictionary to map English tokens to indices
from collections import Counter
en_tokens=[["BOS"]+[tok.text for tok in en_tokenizer.tokenizer(x)] #1
+["EOS"] for x in trainen] #1
PAD=0
UNK=1
word_count=Counter()
for sentence in en_tokens:
for word in sentence:
word_count[word]+=1
frequency=word_count.most_common(50000)
total_en_words=len(frequency)+2
en_word_dict={w[0]:idx+2 for idx,w in enumerate(frequency)} #2
en_word_dict["PAD"]=PAD
en_word_dict["UNK"]=UNK #3
en_idx_dict={v:k for k,v
in en_word_dict.items()} #4
The token-dictionary listing shows EOS appended to English training phrases before their tokens receive indices. section 2.2.1
Listing 12.4 Formatting and padding the tokenized input
import numpy, torch
device="cuda" if torch.cuda.is_available() else "cpu"
text_tokens = numpy.ones((2, 64), dtype=numpy.int32) #1
text_tokens[0, :2] = [token_ids[0], token_ids[-1]] #2
text_tokens[1, :len(token_ids)] = token_ids #3
text_tokens = torch.tensor(text_tokens,
dtype=torch.long, device=device) #4
print(text_tokens)
The formatting listing places start and end IDs in a padded min-DALL-E input. section 12.2.1
