Natural language processing (NLP) uses computation and machine learning to work with human language. It underpins applications that interpret meaning, communication, and text, while inheriting the difficulty of representing ideas that do not have a direct numerical translation. section 2.0 section 3.2.5
NLP connects several layers of language technology:
- Linguistics supplies syntax, semantics, pragmatics, and semiotics for reasoning about meaning and interpretation. section 2.0
- Language modeling progresses from N-grams and naive Bayes through multilayer perceptrons, recurrent neural networks, and LSTMs to transformer-based models. section 2.0
- Embeddings convert text into numbers, but remain approximations because text encodes thoughts, ideas, and communication patterns rather than directly measurable physical quantities. section 3.2.5
Production NLP therefore demands more than deploying a large model. Text creates distinctive operational problems, including imperfect representations and difficult monitoring questions such as how to detect data drift. section 3.2.5 section 2.0
