An NLP pipeline passes natural-language text through processing stages until it produces data, a decision, or a reply. The stages matter because words do not always have one clear meaning: “The chicken is ready to eat” might describe a hungry bird or dinner. A pipeline must turn that ambiguity into something an application can use. section 1.1
For a chatbot, five cooperating subsystems provide a practical design:
- Parse text into structured features. section 1.8
- Analyze features for properties such as sentiment, grammar, and meaning. section 1.8
- Generate candidate replies with templates, search, or language models. section 1.8
- Decide which reply best advances the user’s conversational goal. section 1.8
- Store conversation history and relevant knowledge in a database for later decisions. section 1.8
Not every application needs every subsystem or a deep model. A small intent recognizer can split a message into words, check for a greeting, and choose between two replies. More elaborate pipelines add layers such as part-of-speech tagging and entity relationships; deeper layers can help populate a knowledge base for inference. A useful question is how much processing the task needs, not how many stages can be added. section 1.5.1 section 1.9
Inspect an intermediate result
Run `nlp = spacy.load('en_core_web_sm')`, then `doc = nlp(text)` and `[tok.text for tok in doc]`. The resulting tokens let you inspect an early transformation before using later labels or decisions. Splitting a contraction into separate tokens can help treat “isn’t bad” like “is not bad” in search or sentiment analysis; whether that change is desirable depends on the application. section 2.1.4
Pipelines can also combine different kinds of processing. A question-answering system can join text search, semantic search, and an LLM: fast retrieval narrows the documents, while semantic matching helps find relevant ones. In a generative chatbot, a prompt and conversation memory help shape replies, but fluent output still needs checking: a math-tutor example praises an incorrect answer while giving the correct next number. Design the stages around the task, and inspect both intermediate outputs and final behavior. section 10.3 section 12.5.2
