A search for “machine learning” may surface documents about data scientists and software engineers more often than documents about food service workers. The distributional hypothesis gives that pattern a use: words and phrases appearing in similar contexts tend to have related meanings. In search, those contexts can be the documents, fields, or document sets in which terms occur. section 2.3
Co-occurrence alone is not enough. “Pain” and “ibuprofen” occur together more often than either does with unrelated terms, making their relationship useful to discover. But “the” also appears alongside both—and alongside nearly everything else. Relatedness therefore depends on how distinctive a term’s pattern of co-occurrence is, not just how frequently two terms share documents. A related term need not be a synonym: “pain” and “ibuprofen” are connected without meaning the same thing. section 5.4.4
That principle can feed several search techniques:
- Query matching supplies context: character patterns, nearby phrases, and Boolean combinations each select a document set in which to examine terms. section 2.3
- Embeddings represent patterns of meaning for similarity-based ranking rather than relying only on keyword presence or counts. section 2.3 section 2.7
- Query logs extend the idea beyond documents: queries, clicks, sessions, and users provide behavioral contexts for learning related terms, synonyms, and misspellings. section 2.6
Discover terms related to “advil”
In a health-document collection, start a semantic knowledge graph traversal at the body-field term “advil”. Examine body-field terms appearing at least twice, limit the output to eight, and sort by relatedness. The resulting terms include “motrin”, “aleve”, and “ibuprofen”. This is a ranked set of corpus-specific associations, not a guarantee that every result is interchangeable with the starting term. section 5.4.4
Listing 5.4 Discovering semantically related terms to advil
health_skg = get_skg(engine.get_collection("health"))
nodes_to_traverse = [{"field": "body", #1
"values": ["advil"]}, #2
{"field": "body",
"min_occurrences": 2, #3
"limit": 8}] #4
traversal = health_skg.traverse(*nodes_to_traverse) #5
print_graph(traversal, "advil") #6
The Python listing shows the traversal inputs—starting field and value, minimum occurrences, and result limit—needed to reproduce the related-term lookup. section 5.4.4
