A beginning-of-sequence token marks the start of a sequence and triggers a generative decoder to produce a suitable first token. Common spellings include <SOS> and <s>. section 9.2.3
An end-of-sequence token tells the decoder when it can stop generating. Some datasets use the same marker for both boundaries because the decoder knows when it is starting a generation loop. Choose marker strings that are not ordinary words in the dataset vocabulary. section 9.2.3
>>> SRC = 'en' #1 >>> TGT = 'de' #2 >>> SOS, EOS = '<s>', '</s>' >>> PAD, UNK, MASK = '<pad>', '<unk>', '<mask>' >>> SPECIAL_TOKS = [SOS, PAD, EOS, UNK, MASK] >>> VOCAB_SIZE = 10_000 ... >>> from tokenizers import ByteLevelBPETokenizer #3 >>> tokenize_src = ByteLevelBPETokenizer() >>> tokenize_src.train_from_iterator( ... [x[SRC] for x in sents['train']['translation']], ... vocab_size=10000, min_frequency=2, ... special_tokens=SPECIAL_TOKS) >>> PAD_IDX = tokenize_src.token_to_id(PAD) ... >>> tokenize_tgt = ByteLevelBPETokenizer() >>> tokenize_tgt.train_from_iterator( ... [x[TGT] for x in sents['train']['translation']], ... vocab_size=10000, min_frequency=2, ... special_tokens=SPECIAL_TOKS) >>> assert PAD_IDX == tokenize_tgt.token_to_id(PAD)
The translation setup registers <s> and </s> alongside other special tokens when training separate source and target tokenizers. section 9.2.3



