A handoff summary can sound confident and still mislead an agent. If a claim was paid, a summary saying it was denied invents a problem the user never stated. A one-shot example gives an LLM one sample input and the desired output before the real input, showing what a useful answer should look like. section 12.3.1
Show the kind of handoff you need
For a chat-summary prompt, pair a sample transcript with a short summary: the user requested a representative after failing to validate the member ID needed to find a claim. Then supply the transcript to be summarized. The example demonstrates the focus and phrasing; it does not establish why the next user requested an agent. section 12.3.1
Listing 12.8 A one-shot summarization prompt
<|instruction|> #1 Summarize the following conversation transcript between a user #1 and an AI #1 The summary should be 1-2 sentences long. #1 The summary should emphasize the last task the user was #1 performing that led them to request a human representative. #1 <|transcript|> #2 AI: "Welcome to the automated assistant, how can I help?" #2 User: "Claim status" #2 AI: "Ok, claims. What's your Tax ID?" #2 User: "012345678" #2 AI: "Thank you. What's the member's eight-digit Member ID?" #2 User: "I don't have it" #2 AI: "Ok, let's try something else. What's the member's date #2 of birth?" #2 User: "representative" #2 #2 <|summary|> #2 The user requested a human representative after failing #2 to validate the member ID needed to find the claim. #2 <|transcript|> #3 <... human-readable transcript removed for brevity ...> <|summary|> #4 The user requests a human representative because they #5 want more information about their insurance claim. #5
The prompt separates the instruction, example transcript, and example summary with distinct markers before introducing the next transcript. Clear separation mattered here: using ordinary colon labels produced poorer summaries. section 12.3.1
The same approach can guide structured extraction. A sample transcript paired with JSON helped exclude unwanted fields and duplicated IDs. But the model still changed a key name and returned one numeric ID as a string. Adding explicit instructions for permitted keys and value types produced the desired output. An example is guidance, not a substitute for checking the result. section 12.3.2
Use a one-shot example as a starting point, then test its limits:
- Choose an example close enough to demonstrate the task, but test other conversation flows; a claim-focused example may not cover other kinds of IDs. section 12.3.2
- Check summaries for unsupported claims and verify extracted values against the transcript before relying on them. section 12.3.1 section 12.3.2
- Review generated chatbot utterances rather than accepting them wholesale. An example can encourage variety while still leaving gaps in wording and grammatical forms. section 7.2.3
