A beginning-of-sequence (BOS) token is a special marker placed at the start of a tokenized sentence. It gives an autoregressive decoder an explicit starting point: the decoder reads BOS, predicts the first output token, then continues from the growing sequence until an end-of-sequence (EOS) token signals completion. section 10.1.1 section 9.1.2
In an English-to-French Transformer pipeline, BOS participates in several practical steps:
- Insert BOS before every tokenized phrase and EOS after it, so the model can learn explicit sequence boundaries. section 10.1.1
- Assign BOS an integer through the vocabulary dictionary, alongside PAD and UNK, before converting tokens into model inputs. section 10.1.1
- Begin decoding with BOS, predict one token at a time, and stop when the decoder predicts EOS. section 10.5
- Use BOS and EOS for sentence-based translation, unlike text generation inputs built from a predetermined number of tokens. section 10.1.2
Starting a translation
For the French translation of “I do not speak French,” the decoder first receives BOS. If it predicts “Je,” the next input becomes “BOS Je”; each later prediction extends that sequence until EOS appears. section 9.1.2
Listing 10.2 Mapping English tokens to indexes
from collections import Counter
en=df["en"].tolist() #1
en_tokens=[["BOS"]+tokenizer.tokenize(x)+["EOS"] for x in en] #2
PAD=0
UNK=1
word_count=Counter()
for sentence in en_tokens:
for word in sentence:
word_count[word]+=1
frequency=word_count.most_common(50000) #3
total_en_words=len(frequency)+2
en_word_dict={w[0]:idx+2 for idx,w in enumerate(frequency)} #4
en_word_dict["PAD"]=PAD
en_word_dict["UNK"]=UNK
en_idx_dict={v:k for k,v in en_word_dict.items()} #5
Inspect how BOS and EOS are added around tokenized English phrases before frequency counts and integer vocabulary mappings are created. section 10.1.1
