An embedding is an ordered collection of numbers used to represent one entity as a point or direction in a coordinate space. Each position has a fixed meaning, so order matters: a two-element input can place hardness first and sharpness second. In machine learning, this compact representation lets related values travel through a model together rather than as unrelated scalars. section 2.1 section 1.3.1
Geometrically, an embedding with n elements identifies a point in n-dimensional space. It may also be pictured as an arrow from a reference point: its magnitude is its length, and its direction describes its orientation. The coordinate array depends on the chosen coordinate system, whereas the geometric relation it represents does not. section 2.1.1 section 2.5.5
Feature embeddings
A feature embedding describes properties chosen for a task. A cat-decision input can be written as [hardness, sharpness]. An image classifier can emit a three-element output whose entries correspond, in order, to the probabilities of dog, human, and cat. A document embedding can reserve one coordinate per vocabulary term and store that term’s frequency or TF-IDF value. section 2.1 section 4.1 section 4.6.1
Figure 2.1 Input images and corresponding output vectors denoting probabilities that the image contains a dog and/or human and/or cat, respectively. Example output vectors are shown.
Compare images with their ordered probability-vector outputs to see how coordinates encode model predictions. section 2.1
Embeddings make model operations geometric. A dot product measures how much one vector lies along another; vectors pointing in similar directions tend to have larger dot products, perpendicular vectors have zero dot product, and opposite directions yield a negative value. After dividing by lengths, cosine similarity compares direction rather than scale. section 2.5.6 section 4.6.1
Matrices transform embeddings from one space to another. An m × n matrix maps an n-element vector to an m-element vector, enabling a model to rotate, project, or otherwise reshape feature-space points. Classifiers use such transformations to move examples toward spaces where classes are more clearly separated. section 2.10 section 2.7.3
In code, a vector is a one-dimensional tensor. PyTorch can create one with torch.tensor([...]), access entries by index, and extract subvectors by slicing. Vectors also serve as rows or columns within matrices, while matrices, images, and batches extend the same representation to higher-dimensional tensors. section 2.2.1 section 2.4
An embedding need not be stored densely. A document vector may have one coordinate for every vocabulary word, yet most words are absent from any one document. Storing only present word–value pairs gives a sparse representation that can be expanded when needed. Such vectors support retrieval by comparing a query with archived documents. section 4.1 section 4.6.1
