A token ID is an integer identifier assigned to a token in a vocabulary. It lets a transformer consume tokenized text numerically rather than as raw words or subwords. section 2.2.1 section 12.2.1
A token-ID pipeline connects text, vocabulary entries, and model-ready sequences:
- Tokenization produces words, punctuation, or subwords; a vocabulary maps each distinct token to an integer. section 2.2.1
- A reverse dictionary maps integers back to tokens, allowing an indexed sequence to reconstruct the original phrase. section 2.2.1
- Special IDs mark sequence boundaries and handle batching: BOS or <s> starts a sequence, EOS or </s> ends it, PAD fills shorter sequences, and UNK or <unk> represents unknown tokens. section 2.2.1 section 12.2.1
- The numeric assignments are part of the tokenizer configuration and must agree with pretrained model weights during inference. section 12.2.1
A BPE token-ID sequence
For “panda with top hat reading a book,” tokenization returns [0, 8418, 208, 479, 2583, 4132, 58, 407, 2]. Reversing the vocabulary yields <s>, Ġpanda, Ġwith, Ġtop, Ġhat, Ġreading, Ġa, Ġbook, </s>; Ġ marks a word boundary. section 12.2.1
Listing 12.3 Tokenizing the text prompt
text = "panda with top hat reading a book"
token_ids = tokenizer.tokenize(text) #1
print("Token IDs:", token_ids)
idx2token = {v: k for k, v in vocab.items()} #2
raw_tokens = [idx2token.get(i, "<unk>") for i in token_ids]
print("Raw Tokens:", raw_tokens)
Inspect the tokenizer call and the reverse vocabulary lookup that converts token IDs back into raw subword tokens. section 12.2.1
