An embedding layer converts integer token IDs into dense vectors that carry more information than sparse one-hot encodings. In PyTorch, nn.Embedding can learn these vectors while the surrounding model learns its task. section 7.4.2
Configure an embedding layer by deciding:
- Vocabulary capacity: set num_embeddings to the number of token IDs, including a padding token when one is used. section 7.4.2 section 7.5.1
- Vector width: set embedding_dim to control how many learned values represent each token. section 7.4.2
- Initialization: learn vectors from scratch or load pretrained GloVe vectors, retaining trainability when domain adaptation is needed. section 7.4.3
- Convolution interface: transpose the embedding output when necessary so embedding dimensions become convolution channels rather than the sequence axis. section 7.4.2
Create a trainable embedding table
from torch import nn embedding = nn.Embedding( num_embeddings=2000, embedding_dim=64, padding_idx=0, ) This allocates a 2,000-token vocabulary with 64 values per token and reserves index 0 for padding.
Listing 7.24 Initializing your embedding layer with GloVe vectors
embed = torch.Tensor(embed) #1
print(f'embed.size(): {embed.size()}')
embed = nn.Embedding.from_pretrained(embed, freeze=False) #2
print(embed)
Inspect the GloVe initialization workflow: align vectors with vocabulary order, convert them to a tensor, and load them with nn.Embedding.from_pretrained(..., freeze=False). section 7.4.3
