A chatbot can handle a familiar question with a prepared response, then stumble when a user asks for something no one anticipated. A large language model (LLM) offers another way to respond: it is trained on huge text datasets to predict what word is likely to come next. Repeating that prediction lets it generate longer answers. Its fluency is useful, but it does not establish whether an answer is true. section 1.1.3 section 1.2.1
An LLM can work behind the scenes or respond to users. It can draft dialogue, summarize transcripts, generate training utterances, or answer questions. A hybrid chatbot can keep controlled responses for common intents and route other questions to search or generation. This avoids treating every request as either a fixed script or an unrestricted LLM conversation. section 1.1.3 section 1.2.1
Route an open-ended request
Suppose a customer asks a cake-ordering chatbot for a quick peanut butter pie recipe. The chatbot can recognize the request as a recipe question, retain the complete utterance, combine it with an instruction to act as a helpful kitchen assistant, and call an LLM through a text-generation API. It then checks that the call succeeded before displaying the recipe. The intent controls when generation is used; the LLM handles the variety of dishes users might request. section 2.3 section 2.3.2
Choose the LLM’s job and boundaries before exposing its output:
- For answers grounded in documents, search a knowledge base and give the question and results to the LLM. It can turn ranked results into a concise answer, but wording may vary between runs. section 6.3.1
- For builder assistance, ask for variations on user utterances, then review them. A bare request for password synonyms can produce words that sound wrong for a login problem; adding the customer’s situation produces more relevant suggestions. section 7.1.1 section 7.2.1
- For user-facing generation, test multiple prompts, inputs, models, and parameters. Review the model’s training information where available, screen problematic input, and apply guardrails rather than giving the LLM full control. section 1.2.2 section 1.2.3
A general-purpose LLM is not automatically the best intent classifier. It may cost more, respond more slowly, and produce difficult-to-control output; a purpose-built classifier offers faster, cheaper classification and controlled responses. LLMs can instead help expand scarce intent data or suggest shorter dialogue flows. Check generated suggestions against user needs: a polite insurance-claim flow can still ask for more information than necessary. section 7.1 section 10.1.1
Table 1.3 Dos and don’ts for LLMs
|
Don’t
|
Do
|
Why
|
|---|---|---|
|
Don’t use a model only because you saw it perform well (on a task that you don’t need).
|
Select a model suited to your task, or experiment with several such models.
|
Performance is task-dependent, including any parameters or prompt engineering. Tasks include generation, classification, extraction, question answering, retrieval-augmented generation, summarization, and translation.
|
|
Don’t discard a model or prompt because of one bad experiment.
|
Test on multiple inputs, models, and parameters.
|
Sometimes you’ll get unlucky. It takes multiple tests to have confidence in an LLM configuration.
|
|
Don’t blindly let the LLM have full control, especially in responding to your conversational AI users.
|
Apply guardrails at multiple levels.
|
Use the model-selection checklist to compare task fit, repeated tests, and guardrails before putting generated responses in front of users. section 1.2.3
