A tokenizer segments text into discrete tokens, then maps those tokens to integer IDs that neural networks can process. Depending on the implementation, tokens may be words, punctuation, or subwords. section 2.2.1 section 12.2.1
A practical tokenization pipeline proceeds through these stages:
- Initialize a language-specific spaCy tokenizer, such as the German and English small pipelines. section 2.2.1
- Convert each sentence into tokens and add boundary markers such as BOS and EOS. section 2.2.1 section 2.4.2
- Build a vocabulary dictionary, reserve PAD and UNK IDs, and map frequent tokens to indices. section 2.2.1
- Format IDs for the model by padding sequences or truncating them to a fixed length. section 10.3 section 12.2.1
- Pass the IDs to an encoder when the model needs embeddings rather than raw token numbers. section 10.3 section 12.2
German-to-English translation
The German sentence “Zwei junge weiße Männer sind im Freien in der Nähe vieler Büsche.” becomes word and punctuation tokens, then integer IDs. A trained transformer consumes the resulting sequence and generates English tokens autoregressively until EOS. section 2.2.1 section 2.4.2
Prompt tokenization
A BPE tokenizer converts “panda with top hat reading a book” into IDs for start and end markers plus subword vocabulary entries, then pads the sequence for batched model input. section 12.2.1
import os, spacy
try:
de_tokenizer = spacy.load("de_core_news_sm") #1
except IOError:
os.system("python -m spacy download de_core_news_sm") #2
de_tokenizer = spacy.load("de_core_news_sm")
try:
en_tokenizer = spacy.load("en_core_web_sm") #3
except IOError:
os.system("python -m spacy download en_core_web_sm") #4
en_tokenizer = spacy.load("en_core_web_sm")
Inspect the spaCy setup that loads German and English tokenizer pipelines. section 2.2.1
Listing 12.3 Tokenizing the text prompt
text = "panda with top hat reading a book"
token_ids = tokenizer.tokenize(text) #1
print("Token IDs:", token_ids)
idx2token = {v: k for k, v in vocab.items()} #2
raw_tokens = [idx2token.get(i, "<unk>") for i in token_ids]
print("Raw Tokens:", raw_tokens)
Trace a prompt from raw text through BPE token IDs and vocabulary entries. section 12.2.1
