A beginning-of-sequence token marks where a token sequence starts. In the German-to-English transformer, it is called BOS, or “beginning of sentence.” It gives the decoder an initial token before any English words have been generated. Unlike BOS, EOS marks the end of a sentence; PAD fills shorter sequences in a batch, and UNK represents unknown tokens. section 2.1.2 section 2.2.1
To use BOS in a translation pipeline:
- Tokenize each training phrase, put BOS before its tokens and EOS after them, then include both markers in the token-to-index dictionaries. This is done for the English and German phrases. section 2.2.1
- Encode the German input. Initialize the English decoder input with the English vocabulary’s BOS index, so its first prediction is the token following BOS. section 2.1.2 section 2.4.2
- Append each predicted token to the decoder input and predict again. Use the output for the position after the current input, rather than treating every decoder output as a new token. Stop when the predicted token is EOS. section 2.1.2 section 2.4.2
First steps of a translation
Given the German phrase “Wie geht es dir,” the decoder starts with BOS and might predict “How.” Its next input is BOS How; the next useful prediction is the token following “How.” BOS remains at the front as the generated English sequence grows. section 2.1.2
The marker’s spelling and numeric ID depend on the tokenizer. In min-DALL-E, the start marker is <s>, with ID 0. A tokenized text prompt begins with that ID and ends with </s>, ID 2. Those IDs must match the pretrained weights used for inference. section 12.2.1
Listing 2.2 Creating a dictionary to map English tokens to indices
from collections import Counter
en_tokens=[["BOS"]+[tok.text for tok in en_tokenizer.tokenizer(x)] #1
+["EOS"] for x in trainen] #1
PAD=0
UNK=1
word_count=Counter()
for sentence in en_tokens:
for word in sentence:
word_count[word]+=1
frequency=word_count.most_common(50000)
total_en_words=len(frequency)+2
en_word_dict={w[0]:idx+2 for idx,w in enumerate(frequency)} #2
en_word_dict["PAD"]=PAD
en_word_dict["UNK"]=UNK #3
en_idx_dict={v:k for k,v
in en_word_dict.items()} #4
Shows BOS and EOS being added to English training phrases before the token-to-index dictionary is built. section 2.2.1
Listing 2.9 Defining a function to translate German to English
from utils.transformer_util import subsequent_mask
DEVICE="cuda" if torch.cuda.is_available() else "cpu"
def de2en(ger):
tokenized_ger= [tok.text for tok in de_tokenizer.tokenizer(ger)]
tokenized_ger=["BOS"]+tokenized_ger+["EOS"]
geridx=[de_word_dict.get(i,UNK) for i in tokenized_ger]
src=torch.tensor(geridx).long().to(DEVICE).unsqueeze(0)
src_mask=(src!=0).unsqueeze(-2)
memory=model.encode(src,src_mask) #1
start_symbol=en_word_dict["BOS"]
ys = torch.ones(1, 1).fill_(start_symbol).type_as(src.data)
translation=[]
for i in range(100):
out = model.decode(memory,src_mask,ys,
subsequent_mask(ys.size(1)).type_as(src.data))
prob = model.generator(out[:, -1]) #2
_, next_word = torch.max(prob, dim=1)
next_word = next_word.data[0]
ys = torch.cat([ys, torch.ones(1, 1).type_as(
src.data).fill_(next_word)], dim=1)
sym = en_idx_dict[ys[0, -1].item()]
if sym != 'EOS': #3
translation.append(sym)
else:
break
trans=" ".join(translation)
for x in '''?:;.,'("-!&)%''':
trans=trans.replace(f" {x}",f"{x}") #4
return trans
Shows the translation function obtaining the English BOS index and using it to initialize decoder input. section 2.4.2
