Semantic similarity measures how closely two terms, phrases, documents, or queries correspond in meaning. In a search system, it estimates relatedness from the documents in which terms co-occur, following the distributional hypothesis: terms sharing unusually similar contexts tend to be semantically related. section 5.4.4
A practical relatedness calculation follows this reasoning:
- Collect the documents matching a foreground term or query and compare them with documents matching another term. section 5.4.4
- Reward terms that co-occur in the same documents more often than with random terms. section 5.4.4
- Discount common stop words: frequent terms such as “the” and “a” share contexts broadly but are not treated as meaningfully similar. section 5.4.4
- Use the resulting scores to expand queries, rank related nodes, or select terms that best represent a document’s topic. section 5.4.4 section 5.4.6
From “advil” to related health terms
A health knowledge graph can traverse from “advil” to terms occurring in similar documents. Example scores rank “motrin” at 0.70986, “aleve” at 0.4662, and “ibuprofen” at 0.38264, making the strongest relationships available for search or graph use. section 5.4.4
Content-based recommendation
For a document classified as “star wars,” score its extracted phrases against that topic. Terms such as “luke,” “force,” and “darth vader” score positively, while unrelated terms such as “banana” and “wolverine” score negatively. Keep terms above 0.25, weight them in an expanded query, and search for the five most similar documents. section 5.4.6
Listing 5.4 Discovering semantically related terms to advil
health_skg = get_skg(engine.get_collection("health"))
nodes_to_traverse = [{"field": "body", #1
"values": ["advil"]}, #2
{"field": "body",
"min_occurrences": 2, #3
"limit": 8}] #4
traversal = health_skg.traverse(*nodes_to_traverse) #5
print_graph(traversal, "advil") #6
Inspect the Python knowledge-graph traversal that discovers terms related to “advil” and limits the returned candidates. section 5.4.4
Listing 5.9 Generating a recommendation query from scored phrases
def get_scored_terms(traversal):
return {term: data["relatedness"]
for term, data in traversal["graph"][0]["values"]["star wars"] \
["traversals"][0]["values"].items()}
rec_query = " ".join(f'"{term}"^{score}'
for term, score in get_scored_terms(traversal).items()
if score > 0.25)
print(f"Expanded Query:\n{rec_query}")
Inspect the query-generation step that converts positively scored phrases into weighted search terms. section 5.4.6
