concept NLP pipeline in category nlp

This is an excerpt from Manning's book Natural Language Processing in Action: Understanding, analyzing, and generating text with Python. Login to get full access to this book.

Figure 1.4. Example layers for an NLP pipeline

The NLP pipeline required to build a dialog engine, or chatbot, is similar to the pipeline required to build a question answering system described in Taming Text (Manning, 2013).^[19] However, some of the algorithms listed within the five subsystem blocks may be new to you. We help you implement these in Python to accomplish various NLP tasks essential for most applications, including chatbots.

In chapter 12, you learned how to use all the tools in your NLP toolbox to build an NLP pipeline capable of carrying on a conversation. We demonstrated crude examples of this chatbot dialog capability on small datasets. The humanness, or IQ, of your dialog system seems to be limited by the data you train it with. Most of the NLP approaches you’ve learned give better and better results, if you can scale them up to handle larger datasets.