A sentence-transformer maps sentences or text chunks to fixed-length dense vectors. Semantically similar text lands close in vector space, allowing paraphrases to match even when they share few words. section 4.5.3 section 4.3.5
For retrieval, encode the query into the same space and find nearby vectors. Cosine similarity is a standard measure; dot product suits encoders trained to produce unit-length vectors. section 4.5.3
Choose embedding size with retrieval quality and operating cost in mind:
- 384 dimensions are faster to compute and compare and cheaper to store; all-MiniLM-L6-v2 is a practical default for many enterprise corpora under one million chunks. section 4.5.3
- 768- or 1024-dimensional vectors can preserve more semantic nuance, with higher compute and storage costs. section 4.5.3
Selecting many-shot examples
Encode an example bank once, then compare each query with its vectors to select relevant demonstrations. Precompute and cache bank embeddings in production rather than re-encoding the full bank for every request. section 4.3.5
Listing 4.2 Many-shot prompt assembly (many_shot_demo.py)
# code/chapter04/many_shot_demo.py (key sections -- full source in code/chapter04/many_shot_demo.py)
import json
from pathlib import Path
from sentence_transformers import SentenceTransformer
import numpy as np
EMBEDDER = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
def load_example_bank(path: Path) -> list[dict]:
"""Load a JSONL example bank with `input`, `output`, and optional `category`."""
with open(path) as f:
return [json.loads(line) for line in f if line.strip()]
# Re-encodes the static bank each call for clarity; in production
# precompute and cache these vectors once (or use a vector DB).
def select_by_similarity(bank: list[dict], query: str, k: int) -> list[dict]:
"""Pick the k examples whose `input` is most similar to the query."""
bank_embeddings = EMBEDDER.encode([ex["input"] for ex in bank],
normalize_embeddings=True)
query_embedding = EMBEDDER.encode([query],
normalize_embeddings=True)[0]
scores = bank_embeddings @ query_embedding
# most similar last (recency rule)
top_idx = np.argsort(-scores)[:k][::-1]
return [bank[i] for i in top_idx]
def assemble_prompt(bank: list[dict], query: str, k: int = 50) -> str:
"""Build a many-shot prompt: instruction, k selected examples, then the query."""
selected = select_by_similarity(bank, query, k)
parts = ["Classify the support ticket into one of twelve categories.\n"]
for ex in selected:
parts.append(f"Ticket: {ex['input']}\nCategory: {ex['output']}\n")
parts.append(f"Ticket: {query}\nCategory:")
return "\n".join(parts)See the sentence-transformers implementation for encoding a JSONL example bank and selecting demonstrations by similarity. section 4.3.5
