Embedding layer

As explained in Deep Learning with R, Third Edition

An embedding layer maps discrete integer indices to learned dense vector representations.

hidden_dim <- 64L
inputs <- keras_input(shape = c(max_length), dtype = "int32")
outputs <- inputs |>
  layer_embedding(input_dim = max_tokens,
                  output_dim = hidden_dim,
                  mask_zero = TRUE) |>
  layer_bidirectional(layer_lstm(units = hidden_dim)) |>
  layer_dropout(0.5) |>
  layer_dense(1, activation = "sigmoid")

model <- keras_model(inputs, outputs, name = "lstm_with_embedding")
model |> compile(optimizer = "adam",
                 loss = "binary_crossentropy",
                 metrics = "accuracy")

hidden_dim <- 64

cbow_embedding <- layer_embedding(
  input_dim = max_tokens,
  output_dim = hidden_dim
)

inputs <- keras_input(shape = c(2 * context_size), dtype = "int32")

outputs <- inputs |>
  cbow_embedding() |>
  layer_global_average_pooling_1d() |>
  layer_dense(max_tokens, activation = "softmax")

cbow_model <- keras_model(inputs, outputs)
cbow_model |> compile(
  optimizer = "adam",
  loss = "sparse_categorical_crossentropy",
  metrics = "sparse_categorical_accuracy"
)cbow_model

Model: "functional"

┌─────────────────────────────────┬────────────────────────┬───────────────┐

│ Layer (type)                    │ Output Shape           │       Param # │

├─────────────────────────────────┼────────────────────────┼───────────────┤

│ input_layer_4 (InputLayer)      │ (None, 8)              │             0 │

├─────────────────────────────────┼────────────────────────┼───────────────┤

│ embedding_1 (Embedding)         │ (None, 8, 64)          │     1,920,000 │

├─────────────────────────────────┼────────────────────────┼───────────────┤

│ global_average_pooling1d        │ (None, 64)             │             0 │

│ (GlobalAveragePooling1D)        │                        │               │

├─────────────────────────────────┼────────────────────────┼───────────────┤

│ dense_4 (Dense)                 │ (None, 30000)          │     1,950,000 │

└─────────────────────────────────┴────────────────────────┴───────────────┘

 Total params: 3,870,000 (14.76 MB)

 Trainable params: 3,870,000 (14.76 MB)

 Non-trainable params: 0 (0.00 B)

How Deep Learning with R, Third Edition covers it

How deep it goes

Explains lookup behavior, tensor shapes, trainable weights, and layer arguments, then shows classification and CBOW implementations without deriving embedding mathematics.

Examples

Builds an IMDb sentiment classifier and a CBOW missing-word predictor, then uses character embeddings in a Shakespeare language model.

What it assumes

Introduces integer-to-vector lookup from an intuitive dictionary analogy while worked models rely on integer-tokenized sequences and Keras layers.

What you can do after

Apply it via Keras `layer_embedding()` to integer-token sequences in a recurrent text classifier or character-level language model.

How it connects

Connects embeddings to backpropagation, sequence padding, LSTMs, continuous bag of words, and GRUs.

liveBookmanning.com

Read Deep Learning with R, Third Edition on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Natural Language Processing in Action, Second Editionnn.Embedding is a PyTorch module that maps integer token indices to trainable dense embedding vectors.
• 

  liveBookmanning.com
  How Large Language Models WorkAn embedding layer is a neural-network layer that converts discrete token identities and positions into vector representations.
• 

  liveBookmanning.com
  Time Series Forecasting Using Foundation ModelsAn embedding layer is a neural-network layer that maps input items or tokens to learned numerical vector representations.

Compare how 4 books explain embedding layer →

Related concepts

• embeddingproduces · A numerical vector representation of an item or input that encodes features or relationships useful for computational tasks.
• transformer architecturepart of · A transformer architecture is a neural-network architecture that processes sequences or other structured inputs using attention mechanisms and related transformations.
• autoencoderpart of · An autoencoder is a neural network that encodes input data into an internal representation and decodes it to reconstruct the input or an approximation.
• semanticsused by · Semantics is the meaning and behavior conveyed or assigned to expressions, constructs, representations, or their combinations in a language or symbolic system.
• PyTorchused by · PyTorch is an open-source machine-learning framework for Python that provides tensor computation, automatic differentiation, and tools for building and training neural networks.
• convolutional neural networkpart of · A convolutional neural network is a neural-network architecture that applies learned convolutional operations to extract patterns from structured data.

This explainer was generated by Manning's knowledge graph from the text of Deep Learning with R, Third Edition. The section links point to the parts of the book it draws on.