An end-of-text token is a special token that marks a boundary in tokenized text. In GPT-style preprocessing, <|endoftext|> separates unrelated documents or books concatenated into one training stream, helping the model distinguish their contexts. section 2.4 section 2.9
Implementation uses the token in several related ways:
- Add <|endoftext|> to the vocabulary and map it to an integer token ID; the simple vocabulary example assigns it ID 1130. section 2.4
- Insert the token between independent texts before encoding, so the resulting sequence contains an explicit document boundary. section 2.4
- Use the token as padding when batching variable-length messages; GPT-2 assigns it token ID 50256, and shorter sequences receive repeated padding IDs. section 6.3
- Permit the special token explicitly when encoding with tiktoken, using allowed_special={"<|endoftext|>"}. section 2.5
Separating two texts
Joining "Hello, do you like tea?" and "In the sunlit terraces of the palace." with <|endoftext|> produces a sequence containing the separator ID 1130. Decoding preserves the visible boundary, while unknown words may become <|unk|> with the simple tokenizer. section 2.4
Figure 2.10 When working with multiple independent text source, we add <|endoftext|> tokens between these texts. These <|endoftext|> tokens act as markers, signaling the start or end of a particular segment, allowing for more effective processing and understanding by the LLM.
Shows <|endoftext|> tokens placed between independent text sources as markers for segment boundaries. section 2.4
