concept spell checker in category nlp

This is an excerpt from Manning's book Real-World Natural Language Processing. Login to get full access to this book.

In the final section of this chapter, we will build a practical NLP application—a spell checker—with the Transformer. In the modern world, spell checkers are everywhere. Chances are your web browser is equipped with a spell checker which tells you when you make a spelling mistake by underlining wrong words. Many word processors and editors also run spell checkers by default. Some applications (including Google Docs and Microsoft Word) even point out simple grammatical errors too. Ever wondered how they work? We’ll learn how to formulate this as an NLP problem, prepare the dataset, train, and improve the model below.

We will be using GitHub Typo Corpus[10] as the dataset to train a spell checker. The dataset, created by my collaborator and me, consists of hundreds of thousands of “typo” edits automatically harvested from GitHub. It is the largest dataset of spelling mistakes and their corrections to date, which makes it a perfect choice for training a spell checker.

Figure 8.14: Training a spell checker as an MT system that translates “noisy” sentences into “clean” ones

Listing 8.3: Training a spell checker

fairseq-train \
bin/gtc \
--fp16 \
--arch transformer \
--encoder-layers 6 --decoder-layers 6 \
--encoder-embed-dim 1024 --decoder-embed-dim 1024 \
--encoder-ffn-embed-dim 4096 --decoder-ffn-embed-dim 4096 \
--encoder-attention-heads 16 --decoder-attention-heads 16 \
--share-decoder-input-output-embed \
--optimizer adam --adam-betas '(0.9, 0.997)' --adam-eps 1e-09 --clip-norm 25.0 \
--lr 1e-4 --lr-scheduler inverse_sqrt --warmup-updates 16000 \
--dropout 0.1 --attention-dropout 0.1 --activation-dropout 0.1 \
--weight-decay 0.00025 \
--criterion label_smoothed_cross_entropy --label-smoothing 0.2 \
--max-tokens 4096 \
--save-dir models/gtc01 \
--max-epoch 40