A token ID is the integer assigned to a token in a vocabulary. It lets text or musical events become sequences of integers that a model can process. The ID is a lookup key, not the token itself: a reverse mapping turns a predicted ID back into a word, punctuation mark, or event. section 8.3.1 section 14.1.1
To turn text into token IDs:
- Clean and split the text. Separating punctuation keeps “way.” from being treated as a different token from “way”. section 8.3.1
- Collect unique tokens, assign each an integer, and build both token-to-ID and ID-to-token dictionaries. section 8.3.1
- Look up every token in order to produce the integer sequence used for training. For prompts that may contain unseen words, give an “UNK” token its own ID. section 8.3.1 section 12.2.1
A text vocabulary in practice
In a tokenized novel, frequency ordering assigns the comma ID 0, the period ID 1, and “the” ID 2. The first token, “chapter”, becomes 208. These numbers depend on the vocabulary: another text corpus assigns the period ID 0 and “the” ID 1. section 8.3.1 section 12.2.1
Token IDs also provide training targets. For next-token prediction, divide the integer stream into equal-length inputs and shift it by one token to form outputs. A GPT model then learns, for example, to predict the ID for “old” after the ID for “the”. In translation, English and French vocabularies supply ID sequences that are transformed into embeddings and combined with positional encodings. section 12.1.2 section 10.0
IDs for musical events
Music uses the same idea, but an event has both a type and a value. A note-on event with value 74 has ID 74; a note-off event with value 60 has ID 188 because its ID range begins at 128. A time-shift value of 2 means 0.03 seconds, while its ID lies in the time-shift range beginning at 256. Distinct ranges keep event types apart. section 14.1.1
Listing 8.1 Dictionaries to map tokens to indexes and indexes to tokens
text_length=len(text) #1
num_unique_words=len(words) #2
print(f"the text contains {text_length} words")
print(f"there are {num_unique_words} unique tokens")
word_to_int={v:k for k,v in enumerate(words)} #3
int_to_word={k:v for k,v in enumerate(words)} #4
print({k:v for k,v in word_to_int.items() if k in words[:10]})
print({k:v for k,v in int_to_word.items() if v in words[:10]})
Inspect the two dictionary comprehensions: enumeration assigns token IDs, and swapping keys and values makes generated IDs readable again. section 8.3.1
