An embedding is a dense vector of continuous numbers used to represent an input in a form a neural network can process. For text, the usual starting point is a token index: a word or subword is assigned an integer, then an embedding layer maps that integer to a vector. Learned token embeddings can capture semantic relationships, so tokens used in similar contexts can have nearby vectors. section 8.2 section 8.2.2
From one-hot vectors to compact vectors
In a vocabulary of 12,778 tokens, one-hot encoding represents “happy families are all alike” as a 5 × 12,778 matrix. Using 128-value word embeddings makes it a 5 × 128 matrix. Unlike one-hot vectors, which place every distinct token at the same distance from every other, the learned vectors can reflect similarities in usage. section 8.2.2
In PyTorch, `nn.Embedding(num_embeddings, embedding_dim)` is a trainable lookup table with one vector per index. `num_embeddings` is the vocabulary size; `embedding_dim` is the length of each vector. Its rows start with random values and change during training through backpropagation. Passing a tensor of indexes retrieves the corresponding rows without constructing one-hot vectors. section 8.4.1 section 12.2.2
import math
class Embeddings(nn.Module):
def __init__(self, d_model, vocab):
super().__init__()
self.lut = nn.Embedding(vocab, d_model)
self.d_model = d_model
def forward(self, x):
out = self.lut(x) * math.sqrt(self.d_model)
return out
This small Transformer embedding class shows the lookup directly: `nn.Embedding(vocab, d_model)` returns vectors of length `d_model`, which the class then scales by its square root. section 10.2.1
What happens after lookup depends on the model:
- An LSTM takes token embeddings as sequence input, then uses its recurrent layers to predict the next token. section 8.4
- A Transformer adds positional encoding to token embeddings so its input also carries sequence-order information. In an English-to-French translator, the encoder then produces vectors representing the input phrase’s meaning for the decoder. section 10.2 section 9.1.2
- The same lookup pattern applies beyond prose: a music Transformer embeds musical-event tokens as 512-value vectors before its decoder layers. section 14.1.2
A token embedding is therefore not, by itself, the encoder’s representation of a whole sentence. It supplies compact, learned input vectors; subsequent model layers process those vectors for tasks such as next-token prediction or translation. section 8.4.1 section 9.1.2
