End of text token
An end-of-text token is a special token marking the boundary or termination of a text sequence, document, or generated output.
Build a Text-to-Image Generator (from Scratch)
An end-of-sequence token is a special marker indicating the end of a token sequence.
- How it is used
- Marks sequence completion in tokenized inputs and tells an autoregressive translation decoder when to stop generating.
- How deep it goes
- Shows how to append and index EOS, then use its prediction as a stopping condition; does not detail how the decoder learns that prediction.
- Examples
- Uses German-to-English translation and a tokenized min-DALL-E prompt whose final ID represents </s>.
- What you can do after
- Build it from scratch by adding EOS to translation training phrases and stopping generation when the decoder predicts it.
Read how Build a Text-to-Image Generator (from Scratch) explains end of text token →
Build a Large Language Model (From Scratch)
A special token used to mark the end of a text sequence or response in GPT-style tokenization.
- How it is used
- Marks boundaries between unrelated documents and supplies padding IDs for uniform batches of variable-length messages.
- How deep it goes
- Defines the token, demonstrates vocabulary and tokenizer changes, and shows padding implementation, but does not detail model-side masking.
- Examples
- Uses concatenated sentences, token ID 1130, GPT-2 token ID 50256, tiktoken encoding, and padded spam-message batches.
- What you can do after
- Apply it via a custom tokenizer or tiktoken, insert it between documents, and configure it as a padding token in a PyTorch dataset.
Read how Build a Large Language Model (From Scratch) explains end of text token →
Learn Generative AI with PyTorch
A special end-of-sequence token used to mark completion of decoding or the end of a tokenized sequence.
- How it is used
- Uses EOS to teach a Transformer when an autoregressive translation is complete.
- How deep it goes
- Explains EOS insertion, integer mapping, and decoding termination, but does not detail its loss calculation.
- Examples
- Uses English-to-French phrases, subword pieces, vocabulary dictionaries, and generated translations as working examples.
- What you can do after
- Apply it via a Transformer decoder that stops generation when the predicted token equals EOS.
Read how Learn Generative AI with PyTorch explains end of text token →
Server-Side WebAssembly
An end-of-sequence token is a special model token indicating that generated text should terminate.
- How it is used
- Stops token-by-token story generation when the model emits its designated termination marker.
- How deep it goes
- Explains lookup, fallback behavior, bounded looping, and alternative stopping checks without detailing token-model internals.
- Examples
- Uses the prompt “Once upon a time” and a Rust loop that samples tokens until EOS or a paragraph break.
- What you can do after
- Apply it via Rust token-generation code with EOS lookup and explicit safeguards against runaway output.
Read how Server-Side WebAssembly explains end of text token →
Build a Reasoning Model (From Scratch)
A special token used to mark the end of a document or generated sequence and stop generation.
- How it is used
- Stops streamed inference when the model emits the configured end-of-sequence token.
- How deep it goes
- Implements token-ID lookup, loop comparison, and early termination, without detailing token-training mechanics.
- Examples
- Uses <|endoftext|>, ID 151643, tokenizer.eos_token_id, and a one-sentence language-model prompt.
- What you can do after
- Operate/configure a streamed generator to stop cleanly at the tokenizer’s end-of-sequence ID.
Read how Build a Reasoning Model (From Scratch) explains end of text token →
Related concepts
- special tokenkind of · A special token is a reserved vocabulary item that conveys structural, control, boundary, error, or other auxiliary information rather than ordinary content.
- tokenkind of · A token is a discrete unit of text, such as a word, word fragment, character, or symbol, represented and processed as an element by a language model.
- generationused by · Generation is the computational production of outputs from supplied inputs or context, often by applying a model to produce predictions or new content.
- chat templateequivalent to · A chat template is a formatting specification that serializes conversational messages into the structured input representation expected by a language model.
- LLM wrapperused by · An LLM wrapper is a software layer that mediates access to a large language model through a common interface while handling inputs, outputs, or application-specific controls.
- decoder-only transformeruses · A decoder-only transformer is a Transformer architecture without a separate encoder that processes preceding context to predict or generate subsequent sequence elements.
These guides are generated from the text of Manning books by Manning's knowledge graph and link to the sections they draw on. The full books are available on liveBook and at manning.com.