A model selection playbook is a structured rubric for deciding which model variants are ready for an A/B test. It brings evidence and trade-offs together before teams commit test capacity, engineering resources, and user exposure. The structure can be lightweight or detailed, depending on the need for documentation and alignment. section 7.1
Use the playbook to make the selection decision explicit:
- Define the variants and compare them with a weighted rubric covering offline performance, product heuristics, risk, and strategic alignment. section 7.1 section 7.9
- Compare variants in an employee sandbox before exposing them to users. Capture beta findings and operational readiness checks alongside offline results. section 7.9 section 7.5
- Align stakeholders on objectives and practical details, such as the training data cutoff and expected inference latency. Invite decision-makers into internal testing, and use thoughtful questions to keep feedback useful. section 7.5
- Select variants and document why they should move forward, including known weaknesses and the checks needed during the live test. section 7.5 section 7.9
Set thresholds according to model maturity, use case, product domain, company strategy, and innovation potential. Early-stage models often use experiments to learn whether a design fits the product; mature variants may be tested with a view to shipping them if results support it. Maturity also affects monitoring intensity. section 7.4 section 7.9
An early-stage LLM feature
A first LLM-based product experience may warrant some flexibility on latency, cost, or coverage if it substantially improves user experience. At the same time, raise the bar for safety, evaluation coverage, observability, and rollback readiness. Success of the model family elsewhere does not establish readiness for this product and its users. section 7.4
Carry offline findings into the live experiment. If diversity dropped for a segment offline, track diversity for that segment online with a clear alert threshold. A metric mirror table can pair offline measures with online counterparts, formulas, and alert triggers. section 7.9
Keep the playbook current. Maintain a rolling dataset linking offline results to online outcomes, and update rubric weights as evidence accumulates about which offline measures predict live results. section 7.8.3 section 7.9
