A phantom deployment runs a new AI system beside the existing process while people continue using the established outcomes. The system’s outputs are recorded but ignored for live decisions, allowing comparison without exposing users to its errors. section 8.1
Use the parallel run to:
- Compare new and current outputs over weeks or months. section 8.1
- Find discrepancies, correct problems, and check whether performance degrades over time. section 8.1
- Assess error costs, customer reactions, and safety before replacing an existing function. section 8.1
Customer support chatbot
A company can run an LLM chatbot alongside existing support channels, compare its answers with established responses, and delay replacement until reliability and customer effects are understood. section 8.1
Figure 8.1 When looking at the process diagram, it would seem like replacing FAQs, email tickets, and support numbers could be simplified and streamlined with an LLM-based chatbot. However, the folly of this view is that the process is incomplete. The potential errors and remediation processes required to ensure an LLM will perform accurately are hidden and create more complexity.
Use the process diagram to notice that apparent chatbot simplification hides error handling and remediation work. section 8.1
