6 Building an intelligent chatbot

 

This chapter covers

  • Training an ML model to see similarities between two statements
  • Using deep learning for text classification
  • Integrating a text classification model with a chatbot
  • Training a model to recognize entities in a sentence

Natural language processing (NLP) is a major field in machine learning. Without it, the popular AI chatbots, such as ChatGPT and Copilot would not be possible. While we won’t be building a fully-fledged ChatGPT clone in this chapter, as such a task would be way too vast for us to cover in a single chapter, we will add some intelligent chatbot capabilities to the virtual assistant we’ve built previously.

We will start by introducing a new task to the virtual assistant that will allow the user to see if two sentences have a similar meaning. Then, we will do something to make our whole virtual assistant smarter. The user will no longer have to type very specific words into the console. They will be able to describe the task they want to perform in their own words.

Then we will briefly cover how machine learning models can be trained to recognize specific entities in a sentence. For example, in a sentence such as “Microsoft announced the release of new AI”, a model can be taught to recognize that “Microsoft” is an organization, “AI” is a type of product, and so on.

6.1 Introduction to natural language processing

6.2 Applying sentence similarity

6.2.1 Training a sentence similarity model

6.2.2 Consuming the sentence similarity model

6.2.3 Adding sentence similarity checker to the virtual assistant

6.2.4 Pre-processing sentence similarity data

6.2.5 extracting features from text tokens

6.2.6 Determining the similarity between two sentences

6.3 Applying text classification

6.4 Teaching the chatbot to answer questions

6.4.1 Hallucinations in AI and where they come from

6.5 Named entity recognition technique

6.5.1 Training a NER model with ML.NET

6.5.2 Using the NER model

6.5.3 How NER works

6.5.4 Practical applications of NER

6.5.5 An example of a NER training model

6.6 Project: refining chatbot capabilities

6.6.1 Enhancing the training data

6.6.2 Guarding against hallucinations

6.6.3 Incorporating NER in our chatbot

6.7 Summary