liveBook · Manning

End-of-sequence token

As explained in Build a DeepSeek Model (From Scratch)

An end-of-text token is a special token marking the boundary between documents or text sequences.

An end-of-text token is a special token appended after each document to preserve document boundaries when separate texts are concatenated into one training stream. It gives the model an explicit separator instead of silently merging unrelated sequences. section 6.1.3

Use the tokenization pipeline to:

  • Read the tokenizer’s end-of-text ID through enc.eot_token. section 6.1.3
  • Encode each document, append the ID, and extend a single flat token stream. section 6.1.3
  • Store the resulting IDs as uint16 when the vocabulary fits within that range. section 6.1.3

Document stream construction

For documents A and B, encode A, append <|endoftext|>, then encode B and append the token again. The resulting stream retains both boundaries while remaining suitable for binary training data and memory-mapped storage. section 6.1.3

Listing 6.4 The token encoding function
def encode_corpus(texts: List[str], enc: tiktoken.Encoding) -> np.ndarray:
    """Encodes a list of texts into a single flat stream of token IDs."""
    all_ids = []
    eot_token = enc.eot_token #A
    for text in tqdm(texts, desc="Encoding texts"):
        ids = enc.encode(text)
        all_ids.extend(ids)
        all_ids.append(eot_token) #B
    return np.array(all_ids, dtype=np.uint16) #C
# Inside main():
# 4. Tokenize the datasets
train_ids = encode_corpus([ex['text'] for ex in train_dataset], enc)
val_ids = encode_corpus([ex['text'] for ex in val_dataset], enc)

Inspect the compact encode_corpus implementation that obtains enc.eot_token, appends it after every text, and returns a typed NumPy array. section 6.1.3

How Build a DeepSeek Model (From Scratch) covers it

How deep it goes
Explains retrieval, insertion, storage, and chat-template contrast, but stops short of token-level model internals.
Examples
Uses TinyStories-style text samples, flattened token arrays, and the <|endoftext|> separator.
What it assumes
Assumes familiarity with tokenizers, integer token IDs, and concatenated training datasets.
What you can do after
Apply it via tiktoken by appending enc.eot_token after every encoded document.
How it connects
Connects tokenization, document boundaries, chat templates, autoregressive generation, and end-of-sequence tokens.

Also explained in

Compare how 7 books explain end-of-sequence token →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Build a DeepSeek Model (From Scratch). The section links point to the parts of the book it draws on.