An embedding is a numerical vector representing a token, sentence, document, image, or other input. Its dimensions provide a space in which related inputs can be compared: semantically similar items tend to lie near one another. An embedding model produces a fixed-length vector for an input sequence, making inputs of different lengths usable in retrieval, ranking, and clustering. section 3.4.1 section 3.4
Token embeddings and whole-input embeddings serve different jobs. A learned embedding layer maps each text token ID to a vector; positional information is combined with those vectors before attention. Older methods such as Word2Vec and GloVe assign one vector per word, so “bank” has the same representation in financial and riverside contexts. Transformer-based contextual embeddings instead change with surrounding text. Encoder-only models use bidirectional attention to produce representations suited to understanding and retrieval rather than autoregressive generation. section 2.2.3 section 3.4.1 section 3.3
Encode and compare sentences
With SentenceTransformer('Owen/Owen3-Embedding-0.6B'), model.encode(['Transformers capture context effectively', 'Embeddings help understand context']) produces an array of shape (2, 1024): two fixed-length vectors. For search, encode queries and documents separately, then pass their vectors to util.cos_sim to obtain a query-by-document similarity matrix. The illustrated two-query, two-document result scores one document higher for each query; these scores support ranking, not generated answers. section 3.4.1
In retrieval-augmented generation, documents are embedded and stored in a searchable vector database. A query is encoded with the same model; vector similarity selects relevant documents, which are added to the language model’s prompt. This gives the generator access to external, potentially newer or domain-specific information without retraining it. section 3.4
Embeddings also connect modalities. Text starts with subword token IDs; image encoders can split an image into patches, flatten each patch, and learn a projection into the transformer’s embedding dimension. Positional embeddings retain spatial arrangement. Modality-specific encoders can then project text, image, or audio features into a shared space for cross-modal comparison. This learned alignment is more than concatenating features. section 6.3 section 6.3.1 section 6.2
Figure 6.4 Linear projection W maps patch vectors from an input space ℝdto an embedding space ℝk. Dashed arrows indicate the mapping of example patches through W.
The patch-projection diagram shows how a learned mapping turns raw image-patch vectors into vectors a transformer can process. section 6.3.1
