liveBook · Manning

Beginning-of-sequence token

As explained in Natural Language Processing in Action, Second Edition

A start-of-sequence token is a special token marking the beginning of a sequence and signaling a generative decoder to begin output.

A beginning-of-sequence token marks the start of a sequence and triggers a generative decoder to produce a suitable first token. Common spellings include <SOS> and <s>. section 9.2.3

An end-of-sequence token tells the decoder when it can stop generating. Some datasets use the same marker for both boundaries because the decoder knows when it is starting a generation loop. Choose marker strings that are not ordinary words in the dataset vocabulary. section 9.2.3

>>> SRC = 'en'                             #1
>>> TGT = 'de'                                  #2
>>> SOS, EOS = '<s>', '</s>'
>>> PAD, UNK, MASK = '<pad>', '<unk>', '<mask>'
>>> SPECIAL_TOKS = [SOS, PAD, EOS, UNK, MASK]
>>> VOCAB_SIZE = 10_000
...
>>> from tokenizers import ByteLevelBPETokenizer     #3
>>> tokenize_src = ByteLevelBPETokenizer()
>>> tokenize_src.train_from_iterator(
...     [x[SRC] for x in sents['train']['translation']],
...     vocab_size=10000, min_frequency=2,
...     special_tokens=SPECIAL_TOKS)
>>> PAD_IDX = tokenize_src.token_to_id(PAD)
...
>>> tokenize_tgt = ByteLevelBPETokenizer()
>>> tokenize_tgt.train_from_iterator(
...     [x[TGT] for x in sents['train']['translation']],
...     vocab_size=10000, min_frequency=2,
...     special_tokens=SPECIAL_TOKS)
>>> assert PAD_IDX == tokenize_tgt.token_to_id(PAD)

The translation setup registers <s> and </s> alongside other special tokens when training separate source and target tokenizers. section 9.2.3

How Natural Language Processing in Action, Second Edition covers it

How deep it goes
Defines the marker’s role and selection constraints, and registers it with tokenizers without detailing the generation loop’s start-token implementation.
Examples
Registers <s> and </s> in separate English and German byte-level BPE tokenizers.
What you can do after
Apply it via Hugging Face tokenizers by registering a start marker among special tokens.
How it connects
Links sequence-start markers to end-of-sequence tokens, tokenization, and transformer decoders.

Also explained in

Compare how 4 books explain beginning-of-sequence token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Natural Language Processing in Action, Second Edition. The section links point to the parts of the book it draws on.