liveBook · Manning

Beginning-of-sequence token

Also called BOS token

A beginning-of-sequence token is a special token marking the start of a tokenized sequence, often signaling a generative decoder to begin output.

How 4 Manning books explain it

Natural Language Processing in Action, Second Edition

A start-of-sequence token is a special token marking the beginning of a sequence and signaling a generative decoder to begin output.

How it is used
Uses a start marker to initiate decoder generation in transformer translation.
How deep it goes
Defines the marker’s role and selection constraints, and registers it with tokenizers without detailing the generation loop’s start-token implementation.
Examples
Registers <s> and </s> in separate English and German byte-level BPE tokenizers.
What you can do after
Apply it via Hugging Face tokenizers by registering a start marker among special tokens.

Read how Natural Language Processing in Action, Second Edition explains beginning-of-sequence token →

Build a Text-to-Image Generator (from Scratch)

A beginning-of-sentence token is a special sequence marker that indicates where a generated sentence starts.

How it is used
Starts the decoder’s English output sequence in German-to-English translation.
How deep it goes
Traces BOS from vocabulary construction through decoder initialization and iterative prediction, without specifying its learned representation.
Examples
Uses a German-to-English “Wie geht es dir” translation and a tokenized min-DALL-E text prompt.
What you can do after
Build it from scratch by adding BOS to tokenized phrases and initializing translation decoding with its English vocabulary index.

Read how Build a Text-to-Image Generator (from Scratch) explains beginning-of-sequence token →

Learn Generative AI with PyTorch

The pair of special sequence markers that denote the beginning and end of a tokenized sequence.

How it is used
Starts autoregressive Transformer decoding and marks the boundary before a translated sentence.
How deep it goes
Defines placement, vocabulary indexing, and decoding behavior, but does not detail BOS embedding internals.
Examples
Demonstrates BOS with English-to-French phrases, including “I do not speak French” and the prediction sequence “BOS Je.”
What you can do after
Apply it via a token dictionary and autoregressive decoder to start sequence generation and recognize completion with EOS.

Read how Learn Generative AI with PyTorch explains beginning-of-sequence token →

Rearchitecting LLMs

A BOS token is a beginning-of-sequence marker automatically inserted at the start of a tokenized sequence.

How it is used
Removes the automatically prepended BOS position before plotting token-level activations.
How deep it goes
Defines BOS as tokenizer-inserted and explains its attention-sink behavior in Llama-family models.
Examples
Uses “The cat sleeps” to illustrate heatmaps that exclude BOS from token-level and layer-level views.
What you can do after
Understand why BOS activations are removed before plotting to prevent them from distorting heatmaps.

Read how Rearchitecting LLMs explains beginning-of-sequence token →

What to know first

Related concepts

Questions about beginning-of-sequence token

What is beginning-of-sequence token?

A beginning-of-sequence token is a special token marking the start of a tokenized sequence, often signaling a generative decoder to begin output.

Which Manning books explain beginning-of-sequence token?

Natural Language Processing in Action, Second Edition, Build a Text-to-Image Generator (from Scratch), Learn Generative AI with PyTorch and Rearchitecting LLMs.

What should I know before learning beginning-of-sequence token?

Manning books that teach beginning-of-sequence token usually introduce language model, token, attention and natural language processing first.

What is beginning-of-sequence token also called?

Beginning-of-sequence token is also called BOS token.

These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.