A contextualized representation is a vector whose value depends on the text around the element it represents. A static word embedding assigns one vector to “bank” whether the sentence concerns money or a river. A transformer instead produces different token vectors for those two uses. section 3.4.1
Encoder-only models build these representations using bidirectional attention: a token can attend to words on either side. During masked language modeling, the model uses that context to predict masked tokens. The masked-token prediction head is used for pretraining and removed for fine-tuning; the contextual representations remain useful for downstream tasks. section 3.3.1
The representation’s scale determines how it is used:
- Token vectors distinguish a word’s meaning across contexts, such as the financial and riverside uses of “bank.” section 3.4.1
- Sentence and document vectors can be compared with measures such as cosine similarity for semantic search and retrieval. section 3.4.1
Listing 3.6 Generating sentence embeddings using a pretrained transformer
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('Owen/Owen3-Embedding-0.6B') #1
sentences = [ #2
'Transformers capture context effectively',
'Embeddings help understand context'
]
embeddings = model.encode(sentences) #3
print(embeddings.shape) # output: (2, 1024) #4
The `SentenceTransformer` example encodes two sentences and prints a `(2, 1024)` result: one vector per input sentence. It demonstrates obtaining representations without implementing the transformer. section 3.4.1
Listing 3.7 Similarity search between documents and query
model = SentenceTransformer('Owen/Owen3-Embedding-0.6B') #1
queries = [ #2
'What is an embedding in machine learning?',
'How are embeddings generated in transformer models?'
]
documents = [
'Embeddings are vector representations of data, such as words or
sentences, used to capture semantic relationships in machine learning
models.',
'Transformer models generate embeddings by passing input tokens through
multiple layers of attention and feedforward networks, producing
contextualized vector outputs.'
]
query_embeddings = model.encode(queries, prompt_name='query')
document_embeddings = model.encode(documents)
similarity_scores = util.cos_sim(
query_embeddings, document_embeddings) #3
print(similarity_scores)
The search example encodes queries and documents separately, then applies cosine similarity to their vectors. The resulting scores provide a basis for comparing which documents match each query. section 3.4.1
