Tokenization splits text into words, phrases, subwords, sentences, or other meaningful units. In keyword retrieval, these tokens become terms that an inverted index maps to document IDs for fast lookup. section 3.2 section 4.2
Build a basic tokenization-and-indexing pipeline by:
- Starting with searchable documents. section 3.2
- Splitting each document into tokens such as individual words. section 3.2
- Creating a mapping from each token to the document IDs containing it. section 3.2
- Appending document IDs while iterating over tokens, then retrieving matching documents directly from a token’s list. section 3.2
Two tokenization levels
The sentence “The quick brown fox jumps over the lazy dog” becomes individual word tokens. Sentence-level tokenization instead treats a complete sentence as one unit, enabling a model to produce one fixed-size embedding for the sentence. section 4.2 section 4.2
Listing 4.1 Generate Embeddings on sample data
from sentence_transformers import SentenceTransformer #A model = SentenceTransformer(“all-MiniLM-L6-v2”) #B sentences = ["This is an example sentence.", "Another example sentence."] embeddings = model.encode(sentences) #C print(len(embeddings)) #D
A runnable Sentence Transformers listing connects tokenization to encoding sentences as embedding vectors. section 4.2



































