An embedding turns a token ID into a learned vector of floating-point numbers. Why not use a one-hot vector? One-hot vectors are sparse, have one dimension per vocabulary entry, and treat every token as unrelated to every other token. Dense embeddings use far fewer dimensions and can place related words near one another in a structured space. section 14.5.2 section 14.6
“Movie” and “film”
These words can often replace each other in a sentence. Their one-hot vectors are orthogonal, but their embeddings can be close. That geometric relationship is the point: distance between word vectors can reflect a useful relationship between words. It is learned from data, not guaranteed by the vector format alone. section 14.5.2
A Keras Embedding layer works like a lookup table: each integer token ID retrieves its associated vector. Given input of shape (batch_size, sequence_length), it returns floating-point values of shape (batch_size, sequence_length, embedding_size). Its vectors start at random values and are adjusted through backpropagation for the task at hand. There is no single ideal embedding space for every language task; relationships useful for movie-review sentiment may differ from those useful for legal-document classification. section 14.5.3 section 14.5.3
Listing 14.24 Building an LSTM sequence model with an Embedding layer
hidden_dim = 64 inputs = keras.Input(shape=(max_length,), dtype="int32") x = keras.layers.Embedding( input_dim=max_tokens, output_dim=hidden_dim, mask_zero=True, )(inputs) x = keras.layers.Bidirectional(keras.layers.LSTM(hidden_dim))(x) x = keras.layers.Dropout(0.5)(x) outputs = keras.layers.Dense(1, activation="sigmoid")(x) model = keras.Model(inputs, outputs, name="lstm_with_embedding") model.compile( optimizer="adam", loss="binary_crossentropy", metrics=["accuracy"], )
The movie-review sequence model looks up 64-dimensional token vectors, masks padding with mask_zero=True, and passes the resulting sequence to a bidirectional LSTM. section 14.5.3
Rule of thumb: choose the embedding for the job, then check the whole model. In the movie-review example, a one-hot sequence has shape (600, 30000)—18 million floating-point values per review—and its bidirectional LSTM reaches about 84% test accuracy. Replacing one-hot encoding with a learned Embedding layer makes the token representation much smaller, but the corresponding LSTM still reaches only about 84% test accuracy. A better representation alone does not settle a data-hungry sequence task. section 14.5.1 section 14.5.3
Give an embedding a head start
A Continuous Bag of Words model can learn token vectors without sentiment labels by predicting a missing word from its surrounding words. Its embedding weights can then initialize the movie-review LSTM’s Embedding layer. In the reported comparison, that pretrained model reaches about 89% test accuracy, though simple set-based models remain a strong choice when compute is limited. section 14.5.4 section 14.5.5
Embeddings are not limited to whole words: a character-level language model uses an Embedding layer to assign each input character a 256-dimensional vector. The same distinction still matters—an embedding is the learned numeric representation; the sequence model that consumes it determines how those representations are used. section 15.1.1
