A token ID is an integer assigned to a text token. Tokenizing turns text into a sequence of these IDs, giving a model a numeric input while preserving token order. The mapping can also run backward: look up each ID’s token and join the tokens to recover text. section 15.1
A review as integers
A preprocessed IMDb review begins `[1, 14, 22, 16, ...]`. Each integer stands for a word or a special token; the leading ID marks the start of the review. A word-index lookup can decode the sequence, though the dataset’s reserved IDs require an offset in that lookup. Do not assume an ID has the same meaning in a different vocabulary. section 4.1.1
For ordered text inputs, keep three practical rules in mind:
- Keep IDs in sequence order when the model needs to learn positional dependencies; a sequence model receives the complete ordered token sequence. section 14.5
- Make batch lengths consistent. Truncate long sequences and pad short ones with a special `[PAD]` token; take care that padding does not distort predictions. section 14.5
- Pass IDs through an embedding layer before an LSTM. In the illustrated model, `mask_zero=True` marks zero-valued padding for masking. section 14.5.5
Listing 14.19 Padding IMDb reviews to a fixed sequence length
max_length = 600 max_tokens = 30_000 text_vectorization = layers.TextVectorization( max_tokens=max_tokens, split="whitespace", #1 output_mode="int", #2 output_sequence_length=max_length, #3 ) text_vectorization.adapt(train_ds_no_labels) sequence_train_ds = train_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 ) sequence_val_ds = val_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 ) sequence_test_ds = test_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 )
A Keras `TextVectorization` setup produces fixed-length integer sequences: `output_mode="int"` selects IDs, and `output_sequence_length=600` sets the length used for IMDb review batches. section 14.5
IDs also provide a manageable output for text generation. Rather than classify every possible complete sentence, a language model predicts a distribution over the next token, appends a chosen token, and repeats. The resulting integer sequence can be decoded into text. section 15.1
A token ID is not the embedding itself. In a multimodal Transformer example, ordinary text IDs select from a fixed set of embedding vectors, while image-derived “soft tokens” are vectors supplied directly by an image encoder. section 16.4.2
