10 Natural Language Processing with TensorFlow: Language Modelling

 

This chapter covers,

  • Implementing a TensorFlow data pipeline that can generate inputs and targets from free text for a language modelling task
  • Implementing a GRU based language model
  • Defining and perplexity metric in TensorFlow and understand how to interpret it
  • Training the language model on a text corpus
  • Defining an inference model to generate text based on the trained GRU model
  • Implementing beam-search to uplift the quality of generated text

In the last chapter, we discussed an important NLP task called sentiment analysis. Here we will discuss a new task known as language modelling. Language modelling has been at the heart of natural language processing. Language modelling refers to the task where we predict the next word given a sequence of previous words. For example, given the sentence, I went swimming in the ____, the model would predict the word “pool”. Ground-shattering models like BERT (Bidirectional Encoder Representation from Transformers is a type of transformer based model) are trained using language modelling tasks. This is a prime example of how language modelling can help to actualize innovative models that goes on to be used in a plethora of areas and use cases.

10.1  Processing the data

 
 

10.1.1  What is language modelling?

 
 
 

10.1.2  Downloading and playing with data

 
 
 
 

10.1.3  Too large vocabulary? N-grams to the rescue

 
 

10.1.4  Tokenizing text

 

10.1.5  Defining a tf.data pipeline

 
 

10.2  GRUs in Wonderland: Generating text with deep learning

 
 
 

10.3  Measuring quality of the generated text

 
 
 
 

10.4  Training and evaluating the language model

 
 
 
 

10.5  Generating new text from the language model – Greedy decoding

 
 

10.6  Beam search: Enhancing the predictive power of sequential models

 
 

10.7  Summary

 
 
 

10.8 Answers

 
 
sitemap

Unable to load book!

The book could not be loaded.

(try again in a couple of minutes)

manning.com homepage
test yourself with a liveTest