An embedding represents an item as a vector of numbers that a model can use. For text, this is often more manageable than one-hot encoding: a word vocabulary can make one-hot vectors very wide, while embeddings keep each word’s vector at a chosen size. section 4.5.4 section 4.5.3
The numbers are useful when the model learns a meaningful arrangement. Words used in similar contexts can occupy nearby parts of an embedding space. Simply assigning random numbers gives each word a compact vector, but does not by itself provide that structure. section 4.5.4
# In[]:
embedding_dim = 3
embedding = nn.Embedding(vocab_size, embedding_dim)
example_input = torch.tensor([1, 1, 0, 2])
input_embd = embedding(example_input)
print(input_embd.shape)
input_embd
# Out[]:
torch.Size([4, 3])
tensor([[-2.0552, -0.5413, -0.2367],
[-2.0552, -0.5413, -0.2367],
[ 0.6718, -0.1038, 2.3367],
[-0.0655, 1.0041, 2.0066]], grad_fn=<EmbeddingBackward0>)
Try the PyTorch lookup: `nn.Embedding(vocab_size, 3)` turns four integer indices into a tensor of shape `[4, 3]`. The repeated index returns the same row twice. The lookup table starts with random values and its rows can change during training. section 9.4
A small character model can learn three-number embeddings and pass them through linear layers to predict characters. Plotting its trained vectors in three dimensions shows vowels near other vowels, consonants near other consonants, and a special character apart from them. Three dimensions make the result easy to inspect; larger applications can use far more. section 9.4 section 9.4.1
The same representation idea appears in several settings:
- An image encoder produces an embedding that a text decoder uses to generate a caption; image and caption embeddings are also compared during training. section 2.4
- Categorical-data and recommender-system problems can use embeddings when one-hot vectors are cumbersome; pretrained text embeddings can also be fine-tuned. section 4.5.5
- A diffusion model can embed a timestep and a point’s coordinates before passing them through linear layers to predict noise. section 10.6
A basic learned character embedding is static: a character gets the same vector wherever it appears. Attention adds a way to consider surrounding characters dynamically when context matters. section 9.5
