A model or prompt that looks better during development still needs to prove its value in production. A/B testing compares deployed variants by sending different portions of traffic to each and measuring their outcomes. It turns iteration into a decision informed by observed behavior rather than subjective preference. section 10.6 section 13.1.3
Keep three parts of the workflow distinct:
- Build the alternatives. One model might come from an automated pipeline; another might use a different data-processing step. Prompt variants can likewise be deployed simultaneously. section 10.6 section 13.1.3
- Route and observe production traffic. Langfuse can assign different percentages of traffic to prompt versions and use traces to assess response quality, user satisfaction, and business metrics. section 13.1.3
- Track development runs separately. An MLflow experiment groups runs with parameters, metrics, and artifacts; a Kubeflow experiment groups pipeline runs. Neither grouping, by itself, describes a live traffic split. section 4.1.2 section 5.1.2
Testing a production change
An automated pipeline produces the main model. A second model changes a data-processing step, and the two can be served for an A/B test. For a prompt change, deploy both prompt versions through Langfuse, allocate traffic between them, and compare their traced outcomes. The available workflow describes what to compare, but does not specify a statistical test or sample-size rule. section 10.6 section 13.1.3
