Beginning-of-sequence token
A beginning-of-sequence token is a special token marking the start of a tokenized sequence, often signaling a generative decoder to begin output.
Natural Language Processing in Action, Second Edition
A start-of-sequence token is a special token marking the beginning of a sequence and signaling a generative decoder to begin output.
- How it is used
- Starts decoder generation of a suitable first token in a translation sequence.
- How deep it goes
- Defines the marker, explains its decoder-start role, and shows its registration with tokenizers.
- Examples
- Uses `<SOS>` and `<s>` as marker spellings and registers `<s>` in separate English and German tokenizers.
- What you can do after
- Apply it via Hugging Face tokenizers by registering a distinct start marker for sequence generation.
Build a Text-to-Image Generator (from Scratch)
A beginning-of-sentence token is a special sequence marker that indicates where a generated sentence starts.
- How it is used
- Starts the decoder’s English output sequence in German-to-English translation.
- How deep it goes
- Traces BOS from vocabulary construction through decoder initialization and iterative prediction, without specifying its learned representation.
- Examples
- Uses a German-to-English “Wie geht es dir” translation and a tokenized min-DALL-E text prompt.
- What you can do after
- Build it from scratch by adding BOS to tokenized phrases and initializing translation decoding with its English vocabulary index.
Read how Build a Text-to-Image Generator (from Scratch) explains beginning-of-sequence token →
Learn Generative AI with PyTorch
The pair of special sequence markers that denote the beginning and end of a tokenized sequence.
- How it is used
- Starts autoregressive Transformer decoding and marks the boundary before a translated sentence.
- How deep it goes
- Defines placement, vocabulary indexing, and decoding behavior, but does not detail BOS embedding internals.
- Examples
- Demonstrates BOS with English-to-French phrases, including “I do not speak French” and the prediction sequence “BOS Je.”
- What you can do after
- Apply it via a token dictionary and autoregressive decoder to start sequence generation and recognize completion with EOS.
Read how Learn Generative AI with PyTorch explains beginning-of-sequence token →
Related concepts
- tokenkind of · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
- decoder-only transformeruses · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
- decodeuses · Decoding is the process of transforming an encoded or internal representation into an interpretable or target-form output.
- end of text tokenequivalent to · An end-of-text token is a special token marking the boundary or termination of a text sequence, document, or generated output.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.