Natural language processing (NLP) converts unstructured text into machine-actionable data. In search, it focuses on interpreting queries and documents so systems can retrieve and rank content by language and meaning. section 13.1
Search requires different treatment for documents and queries. Documents usually provide sentences, paragraphs, and surrounding context; queries are often only a few keywords and may combine several ideas. Language detection, part-of-speech detection, phrase detection, and sentiment analysis can therefore be poorly matched to query interpretation when used without additional context. section 2.5.4
Search-oriented NLP supplements linguistic analysis with signals that reveal likely intent:
- Look up query phrases in the document corpus to find their most common domain-specific interpretations. section 2.5.4
- Mine term co-occurrence across previous user sessions to learn intent from similar users. section 2.5.4
- Use embeddings to represent meaning in vectors, allowing queries and documents to be compared semantically rather than only by matching keywords. section 13.1
From greetings to search meaning
“Hello to you!” and “Barev Dzes” express approximately the same greeting meaning despite using different languages. Embeddings place such meanings in a shared vector space, where a system can interpret the prompt and select an appropriate response. The same approach can improve search by representing query and document meaning for retrieval and ranking. section 13.1
Figure 1.1 AI-powered search includes all the technologies and techniques at the intersection of the fields of search and AI. These overlap heavily with and use the fields of data science, machine learning, and deep learning.
Use the overview figure to locate NLP within the overlapping search, artificial intelligence, data science, machine learning, and deep-learning techniques used in AI-powered search. section 1.1
