liveBook · Manning

Static embedding

As explained in LLM Evaluation and Alignment, The Foundational Ideas

A static embedding is a fixed vector representation assigned to a word independently of its surrounding context.

A static embedding assigns each word one fixed vector, regardless of its surrounding context. Cosine similarity compares vector directions, so nearby vectors suggest semantic similarity. section 3.2.2

This representation fails when meaning depends on context: “bank” receives the same vector for financial and river-related sentences. Averaging word vectors also creates a bag-of-words representation that loses order, syntax, and context. section 3.2.2

Use static embeddings when:

  • Computational resources are constrained and a lightweight representation is sufficient. section 3.2.2
  • Interpretability matters and inspecting a word’s fixed neighborhood is useful. section 3.2.2
  • Context-sensitive evaluation is required; prefer contextual embeddings for that need. section 3.2.2

How LLM Evaluation and Alignment, The Foundational Ideas covers it

How deep it goes
Defines fixed vectors and their limitations, but stops short of implementation details for training or encoding them.
Examples
Contrasts the word “bank” across finance and rivers, and compares attorney–lawyer and filed–submitted relationships.
What you can do after
Evaluate/understand semantic similarity using fixed vectors while recognizing failures caused by missing context and word order.
How it connects
Contrasts static embeddings with contextual embeddings and connects averaged vectors to bag-of-words models and cosine similarity.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of LLM Evaluation and Alignment, The Foundational Ideas. The section links point to the parts of the book it draws on.