An embedding represents an item as a learned vector of numbers. For words, it turns token IDs into dense vectors that a model can use. This is useful when one-hot vectors are too large and treat every word as unrelated to every other word: a 600-word review with a 30,000-word vocabulary becomes 18 million one-hot values. section 14.5.1 section 14.6
A word embedding for movie-review classification
A Keras Embedding layer can replace one-hot encoding before an LSTM: max_tokens <- 30000L max_length <- 600L hidden_dim <- 64L inputs <- keras_input(shape = c(max_length), dtype = "int32") outputs <- inputs |> layer_embedding(input_dim = max_tokens, output_dim = hidden_dim, mask_zero = TRUE) |> layer_bidirectional(layer_lstm(units = hidden_dim)) |> layer_dropout(0.5) |> layer_dense(1, activation = "sigmoid") model <- keras_model(inputs, outputs) model |> compile(optimizer = "adam", loss = "binary_crossentropy", metrics = "accuracy") Here, each integer token ID selects a 64-number vector. The embedding output has shape (batch_size, 600, 64); mask_zero = TRUE marks padding so the LSTM can ignore it. With training, this model reached about 83% test accuracy, compared with about 70% for the one-hot LSTM. section 14.5.3 section 14.5.1
Think of an Embedding layer as a lookup table: input_dim sets the range of integer IDs, and output_dim sets each vector’s length. Its vectors start random and change through backpropagation. Unlike sparse, high-dimensional one-hot vectors, learned vectors can place related words near one another. For example, “movie” and “film” should be closer than unrelated words. What counts as useful closeness depends on the task; there is no single ideal word-embedding space for every application. section 14.5.2 section 14.5.3
Embeddings can be learned or reused in several ways:
- Train them with the final model. Backpropagation adjusts token vectors for the prediction task. section 14.5.3
- Pretrain them on unlabeled text. A Continuous Bag of Words (CBOW) model predicts a missing word from nearby words; its learned embedding weights can initialize a classification LSTM. In the movie-review example, this raised test accuracy to about 89%, though the sequence model remained constrained by labeled data. section 14.5.4 section 14.5.5
- Add learned position vectors to token vectors when a Transformer needs word-order information. section 15.3.5
The same idea extends beyond words. An image encoder can produce an image embedding while a prompt encoder produces a vector for a point or box; a mask decoder uses both to predict a segmentation mask. In retrieval-augmented generation, document and query embeddings let a vector database find nearby documents whose text can be added to a prompt. section 11.4.2 section 16.4.3
Figure 14.5 An Embedding layer acts as a dictionary mapping ints to floating-point vectors.
The lookup diagram shows how an integer token ID becomes a floating-point vector. section 14.5.3
