A bag-of-words model counts words but loses their order. An n-gram model restores limited sequence information by estimating which word is likely to follow a fixed-length context of n tokens. It is a simple, efficient statistical baseline—not a complete model of language. section 2.2 section 2.2.1
Use n-gram models with these constraints in mind:
- Choose n to control context: the implementation uses trigrams, while the same approach can train pentagrams or septagrams. section 2.2.1
- Expect frequency-based predictions rather than syntax, semantics, morphology, phonetics, or broader linguistic knowledge. section 2.2 section 2.2.1
- Treat larger context as a trade-off, not an automatic improvement; an enormous n-gram model can be useless. section 2.2.1
Hamlet trigram model
Train a maximum-likelihood trigram model on a Hamlet corpus after padding sentences and generating everygrams. Given up to two preceding tokens, generate a continuation; inspect counts and calculate scores, log scores, entropy, or perplexity. The phrase “To be or not to be” also shows why one fixed n cannot capture every phrase length equally well. section 2.2.1
Listing 2.1 A generative N-grams language model implementation
from nltk.corpus.reader import PlaintextCorpusReader
from nltk.util import everygrams
from nltk.lm.preprocessing import (
pad_both_ends,
flatten,
padded_everygram_pipeline,
)
from nltk.lm import MLE
my_corpus = PlaintextCorpusReader("./", ".*\.txt") #1
for sent in my_corpus.sents(fileids="hamlet.txt"):
print(sent)
padded_trigrams = list(
pad_both_ends(my_corpus.sents(fileids="hamlet.txt")[1104], n=2)
) #2
list(everygrams(padded_trigrams, max_len=3))
list(
flatten(
pad_both_ends(sent, n=2)
for sent in my_corpus.sents(fileids="hamlet.txt")
)
)
train, vocab = padded_everygram_pipeline(
3, my_corpus.sents(fileids="hamlet.txt")
) #3
lm = MLE(3) #4
len(lm.vocab) #5
lm.fit(train, vocab)
print(lm.vocab)
len(lm.vocab)
lm.generate(6, ["to", "be"]) #6
Browse the Python and NLTK implementation for reading a corpus, padding sentences, generating everygrams, preparing training data, and creating an MLE trigram model. section 2.2.1
print(lm.counts)
Lm.counts[["to"]]["be"] #1
print(lm.score("be")) #2
print(lm.score("be", ["to"]))
print(lm.score("be", ["not", "to"]))
print(lm.logscore("be")) #3
print(lm.logscore("be", ["to"]))
print(lm.logscore("be", ["not", "to"]))
test = [("to", "be"), ("or", "not"), ("to", "be")] #4
print(lm.entropy(test))
print(lm.perplexity(test))
Browse model inspection and evaluation examples covering token counts, conditional scores, log scores, entropy, and perplexity. section 2.2.1
