A token ID is an integer assigned to a token during indexing. It lets text enter a model as numeric data; an embedding layer can map each ID to a learned vector. section 14.6 section 14.5.4
Token IDs are useful in several common input patterns:
- Keep IDs in token order when a sequence model needs to learn from word or character order. section 14.5
- Pad or truncate sequences to a shared length for batched computation; check that padding does not distort predictions. section 14.5
- Use IDs as inputs to an embedding layer, which learns a vector representation for each vocabulary token. section 14.5.4
Listing 14.18 Padding IMDb reviews to a fixed sequence length
max_length <- 600
max_tokens <- 30000
text_vectorization <- layer_text_vectorization(
max_tokens = max_tokens,
split = "whitespace", #1
output_mode = "int", #2
output_sequence_length = max_length #3
)
text_vectorization |> adapt(train_ds_no_labels)
sequence_train_ds <- train_ds |>
dataset_map(\(x, y) tuple(text_vectorization(x), y),
num_parallel_calls = 8)
sequence_val_ds <- val_ds |>
dataset_map(\(x, y) tuple(text_vectorization(x), y),
num_parallel_calls = 8)
sequence_test_ds <- test_ds |>
dataset_map(\(x, y) tuple(text_vectorization(x), y),
num_parallel_calls = 8)
See how Keras TextVectorization produces integer sequences of a fixed length for IMDb reviews. section 14.5
Predicting a word from its context
A CBOW model takes eight context token IDs, embeds them, averages their vectors, and predicts the ID of the middle word. section 14.5.4
