A token is one unit in a sequence prepared for a model. A text token may be a word, punctuation mark, or part of a word. Tokenization splits a phrase into those units; converting each unit to a numeric index is a separate step. An embedding then represents the indexed token as a vector. section 2.2.1 section 2.1.1
To prepare text for a transformer, follow the sequence rather than treating a token as raw text:
- Tokenize each phrase. The German phrase “Wie geht es dir” becomes [Wie, geht, es, dir]; a spaCy example separates an English comma and period into their own tokens. section 2.1.1 section 2.2.1
- Build a token-to-index dictionary, then replace tokens with indices for model input. The translation example reserves PAD for padding and UNK for tokens absent from the dictionary, and adds BOS and EOS at phrase boundaries. section 2.2.1
- Turn the indexed sequence into embeddings and retain token order with positional encodings. Self-attention then works on those input representations. section 2.1.1
A padded caption batch
A DistilBERT tokenizer turns “two dogs run” into [CLS], two, dogs, run, [SEP], followed by three [PAD] tokens so it matches the length of “an eagle flies in the sky.” Its attention mask marks the five non-padding positions with 1 and the three padding positions with 0. Thus, a token sequence can include structural markers that are not words from the caption. section 8.2.2
Listing 8.2 Tokenizing captions with the DistilBERT tokenizer
from transformers import DistilBertTokenizer #1
tokenizer = DistilBertTokenizer.from_pretrained(
"distilbert-base-uncased")
encoded=tokenizer(
["two dogs run",
"an eagle flies in the sky"],
padding=True, truncation=True,
max_length=200) #2
print(encoded)
for indexes in encoded['input_ids']:
tokens = tokenizer.convert_ids_to_tokens(indexes)
print(tokens) #3
Run the tokenizer listing to inspect both input IDs and the tokens recovered from them; compare the two captions’ padding. section 8.2.2
Tokens also organize image inputs. A vision transformer divides a 3 × 32 × 32 image into 64 patches and converts each patch into a 48-value vector treated as a token; attention relates those patch tokens. In a text-to-image transformer, VQGAN instead encodes an image as discrete image tokens, while a BART-based path predicts codes from a caption for the VQGAN decoder. These uses share the idea of a sequence, not an identical token format. section 3.3 section 1.2.2
Token limits affect prompts directly. The CLIP text encoder described for Stable Diffusion uses 77 token positions: shorter prompts are padded, longer ones truncated, and each position contributes to a 768-value text representation used to condition image generation. section 10.2.1 section 10.3
