A neural network needs fixed-shaped inputs, but text represented as lists of token IDs can have different lengths. Multi-hot encoding turns each list into a vector with one position per vocabulary term: 1 means the term is present, and 0 means it is absent. The result can be batched and fed directly to a model. section 4.1.2
From a review to a vector
Take “this movie made me cry” and a six-entry vocabulary indexed from 0 to 5. Its token IDs are [0, 1, 3, 0, 5]. Treating them as a set gives {0, 1, 3, 5}, and multi-hot encoding gives [1, 1, 0, 1, 0, 1]. Position 5 records that “cry” appears; position 4 records that “laugh” does not. The repeated 0 still produces just one 1. section 14.4
Two practical ways to produce the vectors are:
- For integer sequences, create a zero-filled NumPy array with shape (number of samples, vocabulary size), then set each sample’s indexed positions to 1.0. With a 10,000-word vocabulary, [8, 5] has ones at positions 5 and 8. section 4.1.2
- For raw text, configure Keras TextVectorization with output_mode="multi_hot", call adapt() on the training inputs to learn a vocabulary, then map the layer over the datasets. A 20,000-term setup produces 20,000 values per review. section 14.4.1
Listing 4.3 Encoding the integer sequences via multi-hot encoding
import numpy as np def multi_hot_encode(sequences, num_classes): results = np.zeros((len(sequences), num_classes)) #1 for i, sequence in enumerate(sequences): results[i][sequence] = 1.0 #2 return results x_train = multi_hot_encode(train_data, num_classes=10000) #3 x_test = multi_hot_encode(test_data, num_classes=10000) #4
The NumPy implementation shows the whole operation: allocate one row per sequence, then mark every ID found in that row. section 4.1.2
A word-level multi-hot vector is an unordered bag of words: it tracks presence, not word order or how often a word appears. That makes it a simple input for a linear sentiment classifier, but it cannot distinguish phrases by their word order. To capture some local order, add bigrams to the vocabulary and multi-hot encode those terms too; the classifier itself need not change. On the IMDb example, the word-only model reaches about 88% test accuracy, versus about 90% with unigrams and bigrams. Longer n-grams require rapidly growing vocabularies, so they are not a straightforward route to modeling long sequences. section 14.4 section 14.4.2
Listing 14.16 Applying a bigram encoding to the IMDb reviews
max_tokens = 30_000 text_vectorization = layers.TextVectorization( max_tokens=max_tokens, split="whitespace", #1 output_mode="multi_hot", ngrams=2, #2 ) text_vectorization.adapt(train_ds_no_labels) bigram_train_ds = train_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 ) bigram_val_ds = val_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 ) bigram_test_ds = test_ds.map( lambda x, y: (text_vectorization(x), y), num_parallel_calls=8 )
Setting ngrams=2 keeps the multi-hot output while allowing unigram and bigram terms into the learned vocabulary. section 14.4.2
