An end-of-text token marks a document boundary during training and lets inference stop when generation reaches that boundary. In the demonstrated Qwen3 workflow, <|endoftext|> has token ID 151643 and is exposed as tokenizer.eos_token_id. section 2.7
Use the token in the generation loop:
- Pass tokenizer.eos_token_id as eos_token_id. section 2.7
Stopping a streamed response
Without an end-token check, a response to “Explain large language models in a single sentence” continues beyond <|endoftext|> into unrelated text. Supplying tokenizer.eos_token_id stops the stream after the intended sentence.
