A token ID is an integer that represents a token in a tokenizer’s vocabulary. Tokenization converts text into IDs that transformer models can process, including words, subwords, punctuation, whitespace, and special tokens. section 5.2.2 section 5.2
Inspect IDs with a Hugging Face tokenizer
from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased") encoded = tokenizer("I am curious", return_tensors="pt") print(encoded["input_ids"]) print(tokenizer.convert_ids_to_tokens(encoded["input_ids"][0])) The ID sequence represents the tokenizer’s tokens. Special tokens such as [CLS] may mark the beginning of the sequence, while subword pieces can use the ## prefix. section 5.2.2
When preparing model inputs, token IDs work with related fields:
- Store IDs under input_ids; the model consumes them during inference. section 3.2.3
- Use padding IDs, often trailing zeros in the shown BERT dataset, to make sequences in a batch the same length. section 5.2.2
- Read attention_mask alongside the IDs so padded positions can be distinguished from actual input tokens. section 5.2.2
