Natural language processing (NLP) uses computers to interpret, analyze, manipulate, or generate human language, especially written text. It matters because text carries much of human communication, knowledge, and cultural production, yet natural language is ambiguous, inconsistent, and constantly changing. section 14.1
From text to a classifier
A practical NLP workflow begins by preprocessing text into numeric input, then building a simple text classifier. This foundation supports more complicated models, including Transformers. The same challenge appears in early machine translation: an IBM and Georgetown system used six hardcoded grammar rules and a lookup table of a few hundred words and suffixes to translate 60 selected Russian sentences accurately. The demonstration worked on its narrow examples, but generalizing beyond them proved difficult because word meanings depend on context and grammar needs countless exceptions. section 14.0 section 14.1
Useful rules of thumb:
- Treat natural language as messy data rather than a fixed formal language. section 14.1
- Expect a system that succeeds on handpicked examples to require substantially more work before it generalizes robustly. section 14.1
- Start with numeric preprocessing and simple text classification before moving toward more complicated models. section 14.0
