A Keras TextVectorization layer turns strings into numbers a model can use. It can split text into words or characters, learn a vocabulary with adapt(), and produce either token IDs or multi-hot vectors. section 14.4.1 section 15.1.1
Encode reviews for sentiment classification
Given labeled IMDb review datasets, create a 20,000-term bag-of-words encoding: max_tokens <- 20000 vectorizer <- layer_text_vectorization( max_tokens = max_tokens, split = "whitespace", output_mode = "multi_hot" ) train_text <- train_ds |> dataset_map(\(x, y) x) adapt(vectorizer, train_text) train_vectors <- train_ds |> dataset_map(\(x, y) tuple(vectorizer(x), y)) Each review becomes a 20,000-number vector recording which vocabulary terms occur. Learn the vocabulary from training text, then apply the same layer to validation and test reviews. section 14.4.1
Listing 14.11 Applying a bag-of-words encoding to the IMDb reviews
max_tokens <- 20000
text_vectorization <- layer_text_vectorization(
max_tokens = max_tokens,
split = "whitespace", #1
output_mode = "multi_hot"
)
train_ds_no_labels <- train_ds |> dataset_map(\(x, y) x)
adapt(text_vectorization, train_ds_no_labels)
bag_of_words_train_ds <- train_ds |>
dataset_map(\(x, y) tuple(text_vectorization(x), y),
num_parallel_calls = 8)
bag_of_words_val_ds <- val_ds |>
dataset_map(\(x, y) tuple(text_vectorization(x), y),
num_parallel_calls = 8)
bag_of_words_test_ds = test_ds |>
dataset_map(\(x, y) tuple(text_vectorization(x), y),
num_parallel_calls = 8)
The full listing applies one adapted layer to training, validation, and test datasets while keeping their labels. section 14.4.1
For a little word order, set ngrams = 2 and increase max_tokens to 30,000. The vocabulary can then contain both single words and two-word phrases; the same linear classifier works with the wider vectors. On the IMDb test set, this change raised accuracy from about 88% to 90%. Longer n-grams quickly demand much larger vocabularies. section 14.4.1 section 14.4.2
For next-character prediction, use split = "character" instead. Set standardize = NULL to retain case and punctuation, and output_sequence_length to fix each integer sequence’s length. A Shakespeare example learns a 67-character vocabulary before feeding token IDs to an embedding and GRU. section 15.1.1
Translation needs separate vocabularies for its two languages. An English-to-Spanish pipeline uses two layers with integer output and custom Spanish standardization that preserves [start] and [end] while removing punctuation, including ¿. It then offsets Spanish token IDs to form next-token targets and masks padded labels. section 15.2.1
Listing 15.9 Learning token vocabularies for English and Spanish text
punctuation_regex <- r"---([!"#$%&'()*+,./:;<=>?@\\^_`{|}~¡¿-])---"
library(tensorflow, exclude = c("set_random_seed", "shape"))
custom_standardization <- function(input_string) {
input_string |>
tf$strings$lower() |>
tf$strings$regex_replace(punctuation_regex, "")
}
input_string <- as_tensor("[start] ¡corre! [end]")
custom_standardization(input_string)
tf.Tensor(b'[start] corre [end]', shape=(), dtype=string)
vocab_size <- 15000
sequence_length <- 20
english_tokenizer <- layer_text_vectorization(
max_tokens = vocab_size,
output_mode = "int",
output_sequence_length = sequence_length
)
spanish_tokenizer <- layer_text_vectorization(
max_tokens = vocab_size,
output_mode = "int",
output_sequence_length = sequence_length + 1,
standardize = custom_standardization
)
adapt(english_tokenizer, train_pairs$english)
adapt(spanish_tokenizer, train_pairs$spanish)
The listing shows how separate vocabulary sizes, sequence lengths, and standardization choices are configured for translation. section 15.2.1
