Beginning-of-sequence token
A beginning-of-sequence token is a special token marking the start of a tokenized sequence, often signaling a generative decoder to begin output.
Natural Language Processing in Action, Second Edition
A start-of-sequence token is a special token marking the beginning of a sequence and signaling a generative decoder to begin output.
- How it is used
- Uses a start marker to initiate decoder generation in transformer translation.
- How deep it goes
- Defines the marker’s role and selection constraints, and registers it with tokenizers without detailing the generation loop’s start-token implementation.
- Examples
- Registers <s> and </s> in separate English and German byte-level BPE tokenizers.
- What you can do after
- Apply it via Hugging Face tokenizers by registering a start marker among special tokens.
Build a Text-to-Image Generator (from Scratch)
A beginning-of-sentence token is a special sequence marker that indicates where a generated sentence starts.
- How it is used
- Starts the decoder’s English output sequence in German-to-English translation.
- How deep it goes
- Traces BOS from vocabulary construction through decoder initialization and iterative prediction, without specifying its learned representation.
- Examples
- Uses a German-to-English “Wie geht es dir” translation and a tokenized min-DALL-E text prompt.
- What you can do after
- Build it from scratch by adding BOS to tokenized phrases and initializing translation decoding with its English vocabulary index.
Read how Build a Text-to-Image Generator (from Scratch) explains beginning-of-sequence token →
Learn Generative AI with PyTorch
The pair of special sequence markers that denote the beginning and end of a tokenized sequence.
- How it is used
- Starts autoregressive Transformer decoding and marks the boundary before a translated sentence.
- How deep it goes
- Defines placement, vocabulary indexing, and decoding behavior, but does not detail BOS embedding internals.
- Examples
- Demonstrates BOS with English-to-French phrases, including “I do not speak French” and the prediction sequence “BOS Je.”
- What you can do after
- Apply it via a token dictionary and autoregressive decoder to start sequence generation and recognize completion with EOS.
Read how Learn Generative AI with PyTorch explains beginning-of-sequence token →
Rearchitecting LLMs
A BOS token is a beginning-of-sequence marker automatically inserted at the start of a tokenized sequence.
- How it is used
- Removes the automatically prepended BOS position before plotting token-level activations.
- How deep it goes
- Defines BOS as tokenizer-inserted and explains its attention-sink behavior in Llama-family models.
- Examples
- Uses “The cat sleeps” to illustrate heatmaps that exclude BOS from token-level and layer-level views.
- What you can do after
- Understand why BOS activations are removed before plotting to prevent them from distorting heatmaps.
Read how Rearchitecting LLMs explains beginning-of-sequence token →
What to know first
- language modelA language model is a computational model that represents patterns in linguistic sequences to assign probabilities, predict, generate, or evaluate language.
- tokenA token is a discrete unit of text, such as a word, subword, punctuation mark, or symbol, produced by tokenization and processed by a language model.
- attentionAttention is a neural-network mechanism that computes input-dependent relevance weights to combine information from representations for a context-dependent computation.
- natural language processingNatural language processing is the field of computing concerned with computational methods for analyzing, interpreting, transforming, and generating human language in text or speech.
Related concepts
- attention sinkuses · An attention sink is a sequence position that attracts attention from many queries and serves as a stable shared target during attention computation.
- decoder-only transformeruses · A decoder-only transformer is a Transformer language model with decoder layers and no separate encoder, generating sequence elements autoregressively from preceding context.
- decoding strategyuses · A decoding strategy is a procedure for converting an encoded or internal representation into a target output, including selecting tokens during sequence generation.
- end-of-sequence tokenequivalent to · An end-of-sequence token is a special token that marks the end of a token sequence or document and may signal that generation should stop.
Questions about beginning-of-sequence token
What is beginning-of-sequence token?
A beginning-of-sequence token is a special token marking the start of a tokenized sequence, often signaling a generative decoder to begin output.
Which Manning books explain beginning-of-sequence token?
Natural Language Processing in Action, Second Edition, Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch and Rearchitecting LLMs.
What should I know before learning beginning-of-sequence token?
Manning books that teach beginning-of-sequence token usually introduce language model, token, attention and natural language processing first.
What is beginning-of-sequence token also called?
Beginning-of-sequence token is also called BOS token.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.



