liveBook · Manning

Token id

As explained in Grokking AI Algorithms, Second Edition

A token ID is an integer assigned to a token so that the token can be represented numerically for model processing.

A token ID is an integer assigned to a token in a vocabulary. It lets text be represented as numbers that a model can process. section 11.4 section 11.9

A token-to-ID mapping

In the example vocabulary, `th` maps to 24 and `in` maps to 8. Replacing text tokens with these IDs creates a numerical token stream. section 11.9 section 11.10

Table 11.6 The vocabulary based on the training data

ID

Token

 

ID

Token

 

ID

Token

0

a

 

12

o

 

24

th

1

l

 

13

v

 

25

k

2

ic

 

14

er

 

26

,

3

e

 

15

y

 

27

:

4

w

 

16

ti

 

28

c

5

s

 

17

re

 

29

r

6

b

 

18

d

 

30

p

7

g

 

19

f

 

31

u

8

in

 

20

i

 

32

"

9

n

 

21

h

 

33

?

10

ing

 

22

ers

 

 

 

11

t

 

23

on

 

 

 

Browse the vocabulary table to see token IDs paired with their tokens. section 11.9

How Grokking AI Algorithms, Second Edition covers it

How deep it goes
Defines IDs with a vocabulary table and traces them into token streams and batches, without detailing ID assignment beyond the example.
Examples
Shows BPE-derived vocabulary entries such as `th` and `in` mapped to integers, then represents training text as an ID stream.
What it assumes
Teaches vocabulary-based encoding from text, rather than requiring prior knowledge of how text becomes numerical model input.
What you can do after
Apply token-to-ID mappings to encode text as numerical sequences and divide those sequences into context-window-sized batches.
How it connects
Connects token IDs to tokenization, vocabulary, vectorization, token streams, context windows, and training batches.
Cover of Grokking AI Algorithms, Second Edition
liveBookmanning.com
Read Grokking AI Algorithms, Second Edition on liveBook Get the book at manning.com

Also explained in

Compare how 7 books explain token id →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Grokking AI Algorithms, Second Edition. The section links point to the parts of the book it draws on.