Similarity search finds the closest matches to a query in a collection, then ranks them by relevance. A chatbot can use it to retrieve a reply or a passage that helps answer a question. The crucial choice is what “similar” means: shared characters and words, or similar meaning. section 1.9 section 10.3.3
Choose a representation that fits the mismatch you need to handle:
- Character edit distance can match an incoming statement to a stored statement–response pair without tokenization or feature generation. It is a simple option when you have example conversations. section 1.9
- Full-text search indexes words or character trigrams. Trigrams help with spelling differences, but matching spellings alone can miss documents that use different words for the same idea. section 4.8.1 section 10.3.1
- Semantic search compares vectors intended to represent meaning. Topic vectors from LSA or LDiA and learned embeddings let a query match relevant text even when its exact words are absent. section 4.8.1 section 10.3.2
Finding an FAQ answer
Fit a TF–IDF vectorizer on stored questions, transform those vectors with truncated SVD, and transform a new question through the same steps. Compare its topic vector with the stored question vectors and return the answer associated with the highest-scoring question. In the FAQ example, “decrease overfitting” retrieves a question phrased as “reduce overfitting”—a match the earlier TF–IDF approach struggled with. A small collection can be checked question by question. section 4.9
Listing 4.13 Creating an answer function for the bot that uses semantic similarity
>>> vectorizer = TfidfVectorizer()
>>> vectorizer.fit(df['question'])
>>> tfidfvectors = vectorizer.transform(df['question'])
>>> svd = TruncatedSVD(n_components=16, n_iterations=100)
>>> tfidfvectors_16d = svd.fit_transform(tfidfvectors)
>>>
>>> def bot_reply(question):
... question_tfidf = vectorizer.transform([question]).todense()
... question_16d = svd.transformnp.asarray(question_tfidf)
... idx = question_16d.dot(tfidfvectors_16d.T).argmax()
... print(
... f"Your question:\n {question}\n\n"
... f"Most similar FAQ question:\n {df['question'][idx]}\n\n"
... f"Answer to that FAQ question:\n {df['answer'][idx]}\n\n"
... )
Inspect the Python answer function for the vectorization, SVD transformation, similarity calculation, and selection of the top FAQ match. section 4.9
For embedding-based search, encode documents and the query in the same vector space, score their similarity, and rank the results. Comparing a query against every dense document vector becomes costly as the collection grows—especially when documents are split into shorter passages. Approximate-nearest-neighbor indexing and vector databases help retrieve candidates without checking every vector. Unlike sparse word tables, dense semantic vectors do not fit traditional inverted indexes easily. section 10.3.3 section 4.8.1 section 10.3.7
Retrieval is also a building block for question answering: find relevant passages, then pass them to a reader that extracts an answer or a generator that composes one. Keep retrieval and answering distinct; a plausible generated answer is not proof that the needed information was found. Search systems can combine fast full-text matching with semantic matching to improve the candidates they supply. section 10.3 section 10.3.9 section 10.3.11
