An AI/ML model development workflow turns a product need into an evaluated, deployable, and improvable system. It begins with a defined user problem, success criteria, and constraints rather than with model training alone. Offline evaluation estimates whether a change meets those objectives before exposing it to users, while later online testing checks performance in a live environment. section 2.1.5 section 1.3
After each model iteration, teams choose the next step using metrics, confidence, urgency, and product requirements:
- Re-run offline evaluations when more iteration or analysis is needed. section 2.1.5
- Release to online testing when changes appear sufficiently promising. section 2.1.5
- Retrain when new or additional data is available or required. section 2.1.5
- Deploy to production when validation shows the model is ready, or pause for a strategy change. section 2.1.5
Evaluation supports the workflow but does not choose its objective. Product requirements, user research, domain expertise, business goals, safety requirements, and engineering constraints define what success means. For agents, planning also includes permitted tasks, available tools, required constraints, escalation conditions, and unacceptable failure modes. section 2.1.5
Movie recommendations
A streaming service can evaluate a recommendation model offline before user exposure. Precision at K suits a list such as “Dramatic Yet Light Movies,” where the top results should be highly relevant. Recall suits “Your Recent Shows,” where retrieving all relevant past shows supports a complete experience. Promising offline results can lead to online testing; weak or inconclusive results can trigger iteration instead. section 1.3 section 2.1.5
Figure 2.2 flow captures how evaluation results can guide decisions that influence a team’s roadmap and time spent. Teams often go through multiple rounds of offline evaluation before moving to online testing. After each development cycle, the next step can vary, whether it's rerunning evaluations, retraining the model, or shifting to the online evaluation phase.
Use the flow to inspect how repeated offline evaluation, online testing, retraining, deployment, and strategy changes shape the development roadmap. section 2.1.5

