An embedding is a fixed-length vector that lets a model work with a token, image patch, or whole input numerically. Token embeddings are dense, learned representations: unlike sparse one-hot encodings, they can place semantically similar words near one another. The number of values in each vector is its embedding dimension. section 2.2.3 section 2.1.2
For text, start by tokenizing a sentence and converting each token to an index. An embedding layer maps each index to a learned vector. Four German tokens, each mapped to 256 values, produce a 4 × 256 matrix. A transformer also needs token order, so it adds a same-sized positional encoding to these vectors. The sum is the input embedding—not the word embedding alone. section 2.1.1 section 2.1.2 section 2.2.3
from torch import nn
import math
class Embeddings(nn.Module):
def __init__(self, d_model, vocab):
super().__init__()
self.lut = nn.Embedding(vocab, d_model)
self.d_model = d_model
def forward(self, x):
out = self.lut(x) * math.sqrt(self.d_model)
return out
This PyTorch implementation makes the token-index lookup explicit: nn.Embedding stores a vocabulary-sized set of learned vectors, and forward scales the selected vectors by the square root of the embedding dimension. section 2.2.3
The same pattern extends to images, but the units change. A vision transformer divides an image into patches, converts each patch to a vector, and adds positional information before processing the sequence. For a CIFAR-10 classifier, 64 patches plus a class token form 65 vectors of 48 values each. In an image-captioning transformer, a linear layer embeds image patches; an encoder processes them, and a decoder uses the resulting image features to generate caption tokens through cross-attention. section 3.1.1 section 4.3.1 section 4.3.2
An embedding can also summarize an entire input rather than one token or patch. CLIP places text and images in a shared space with same-dimensional embeddings. Its text encoder uses the final-layer representation of a summary token; training brings embeddings of matching text–image pairs closer and pushes nonmatching pairs apart. Given a prompt, compare its embedding with candidate image embeddings using cosine similarity, then rank the images. Candidate embeddings can be saved for later searches. section 8.3.1 section 8.1 section 8.1.2 section 8.4.2
Two ways to use the vectors
“Wie geht es dir” becomes four token vectors for a German-to-English transformer. “A dog plays on the beach” becomes one text embedding that can be compared with embeddings of 8,091 Flickr 8k images. The first supports sequence processing; the second supports whole-input matching. section 2.1.1 section 8.1.2
Keep an embedding distinct from the output it helps produce. In latent diffusion, a prompt’s text embedding conditions iterative denoising; a VAE decoder turns the resulting latent image into a full-resolution image. The embedding guides generation—it is not itself the generated image. section 1.3.2
