A token ID gives a model a number, but not a useful measure of meaning. An embedding is a vector representation that adds learned context to tokenized data. Embeddings make comparison possible: vectors produced by the same model in the same dimensional space can be compared by distance. That makes them useful for searching documents or matching customer questions without hosting a large generative model. section 4.4.2 section 2.3.1
The meaning is an approximation learned from usage, not a direct translation of language. Words that appear in similar contexts can acquire related representations. Word2Vec’s familiar illustration is that the vector for “king,” minus “man,” plus “woman,” has “queen” as a nearest neighbor. That relationship is striking, but it does not give the vector human pragmatic understanding: “I need food” does not, by itself, establish that someone is in danger. section 2.2.5 section 3.2.5
One way to learn word embeddings is continuous bag-of-words (CBoW). It uses surrounding words to predict a word in the middle of a context window; for “Learning about linguistics makes me happy,” the surrounding words can be used to guess “linguistics.” This uses context on both sides of the target, although predictions remain shaped by patterns seen during training. In a transformer encoder, an embedding layer represents tokens before self-attention and positional encoding help account for where they occur. section 2.2.4 section 2.3.1
Listing 4.11 Example embeddings
import numpy as np
from sentence_transformers import SentenceTransformer
from datasets import load_dataset
model_ckpt = "sentence-transformers/all-MiniLM-L6-v2" #1
model = SentenceTransformer(model_ckpt)
embs_train = load_dataset("tweet_eval", "emoji", split="train[:1000]")
embs_test = load_dataset("tweet_eval", "emoji", split="test[:100]")
def embed_text(example): #2
embedding = model.encode(example["text"])
return {"embedding": np.array(embedding, dtype=np.float32)}
print(f"Train 1: {embs_train[0]}")
embs_train = embs_train.map(embed_text, batched=False)
embs_test = embs_test.map(embed_text, batched=False)
embs_train.add_faiss_index("embedding") #3
#
idx, knn = 1, 3 # Select the first query and 3 nearest neighbors #4
query = np.array(embs_test[idx]["embedding"], dtype=np.float32)
scores, samples = embs_train.get_nearest_examples("embedding", query, k=knn)
print(f"QUERY LABEL: {embs_test[idx]['label']}") #5
print(f"QUERY TEXT: {embs_test[idx]['text'][:200]} [...]\n")
print("=" * 50)
print("Retrieved Documents:")
for score, label, text in zip(scores, samples["label"], samples["text"]):
print("=" * 50)
print(f"TEXT:\n{text[:200]} [...]")
print(f"SCORE: {score:.2f}")
print(f"LABEL: {label}")
A compact Python path from text to retrieval: encode dataset examples with SentenceTransformer, store their embeddings, add a FAISS index, and query nearest neighbors. section 4.4.2
For a practical search system, check that stored items and incoming queries use the same embedding model and dimensional space. Embeddings can support fast retrieval and retrieval-augmented generation, but closeness is not a guarantee of correctness. Data quality and changing language still matter, and correctness or drift in text can be difficult to measure. When selecting a model for a new task, consider whether retrieval with a smaller embedding model solves the problem before building around a much larger LLM. section 4.4.2 section 3.2.5 section 4.6
The same comparison idea extends beyond text. Systems such as ImageBind place different data types in a shared embedding space, opening up searches across modalities—for example, looking for images of animals whose sounds resemble an uploaded audio clip. Shared spaces can also help organize text, image, audio, and video datasets for search and recommendations. section 12.2.3 section 12.2.4
