concept idea in category nlp

appears as: idea, n idea, ideas, The idea
Getting Started with Natural Language Processing

This is an excerpt from Manning's book Getting Started with Natural Language Processing.

This type of prediction based on the sequences of words and characters in context is known as language modeling, and it is at the core of text prediction on your smartphone, or smart reply technology in your email. It is based on the idea that the sequences of plausible words can be learned in a statistical way – that is, calculating the probabilities on some real data. The data is of a paramount importance: both quality and quantity matter. The large quantity of data that became available to the algorithms in the recent decades is one of the strongest catalysts that allowed NLP to move forward and achieve such impressive results. Another is the quality: if you use text prediction on your phone a lot, you might notice that if you had a friend or a colleague called Lang and used this surname quite a lot in messages, your phone will ultimately learn to offer this as the most probable suggestion even if it wasn’t part of the original set. That is, your phone adapts to your personal vocabulary and uses your own data to do that.

Another factor that contributed to the success of conversational agents and predictive text technology is the development of algorithms themselves. This description would not be complete without mentioning that modern text prediction and chatbots increasingly use neural-based language modeling: the idea is the same – the algorithm still tries to predict the next character or word based on context, but the use of context is more flexible. For instance, it might be the case that the most useful information for the next word prediction is not 2 characters or words away, but in the beginning of the sentence or in the previous phrase. The n-gram models are really limited in this respect, whereas more computationally expensive approaches such as neural models can deal with encoding large amounts of information from wider context and even identify which bits of the context matter more.

Information search is based on the idea that the content of a document or set of documents is relevant given the content of a particular query, so both documents and queries data structures should keep the contents of all documents and all queries.