Natural language processing (NLP) lets software work with the words people use, rather than requiring people to speak in commands designed for computers. Natural language is harder to process than a programming language: a word or phrase can have more than one interpretation, and its meaning can depend on context. An NLP system may analyze text, generate a reply, or do both. section 1.1 section 1.2 section 1.2.3
Common applications turn that capability into specific jobs:
- Search can use the meaning of a document, not just its words; a chatbot can use search to retrieve a relevant response. section 1.3 section 1.12
- Sentiment analysis, spelling correction, autocomplete, and grammar checking help classify or improve writing. section 1.3
- Text generation can produce short assistant replies, longer reports, or suggestions for programming code. section 1.3 section 1.3.1
One route from words to computation is to extract numerical vectors from text. Statistics and machine learning can then use those vectors to look beyond word counts toward semantic analysis. Different approaches capture different amounts of context: bag-of-words models provide a simpler baseline, while transformer models can perform better on some tasks. Convolutional neural networks offer another approach, detecting patterns in sequences of neighboring words and supporting text classification. section 1.2.3 section 7.0
A comparison that needs context
For sarcasm detection in an isolated Twitter message, a bag-of-words model achieved 63% accuracy and a transformer model 81%. Those results do not mean conversation is solved: people remain better at recognizing sarcasm across an ongoing dialogue, where context matters. Test an NLP system against the setting in which it will actually be used. section 1.2.3
// Determine whether the sentiment of text is positive
// Use a web service
async function isPositive(text: string): Promise<boolean> {
A short TypeScript comment and function signature illustrate a code-generation prompt: the developer states the intended task, and a code assistant suggests an implementation. Generated code still warrants scrutiny, including for assumptions inherited from the system that produced it. section 1.3.1
NLP outputs are not inherently reliable or neutral. Pipelines can make mistakes and carry bias, even when their writing sounds fluent. Treat a useful result as something to check, especially before using it to answer people or make decisions that affect them. section 1.3 section 1.0
