Model selection is the decision about which evaluated model or combination of models to use in production. Testing produces evidence; selection weighs that evidence against system requirements and records a choice that stakeholders can understand and review. The highest score on one test is rarely enough when models serve a larger application. section 8.0 section 8.3
Treat selection as a delivery task with explicit checkpoints:
- Gather test results, how the tests were conducted, and the system requirements. Judge both what each result says and how informative it is. section 8.3
- Compare functional performance with nonfunctional requirements, including latency, throughput, memory footprint, cost, and carbon impact. A strong functional result does not rescue a model that cannot meet a required constraint. section 8.2.7
- Choose a comparison method suited to the project. Quantitative options include rankings, multi-criteria decision making, and Pareto fronts; qualitative considerations can also affect the choice. section 8.6
- Document the models chosen, how components will be combined, and the reasons for the decision. Review the selection with the customer and obtain sign-off. If the evidence cannot clearly distinguish two candidates, record that uncertainty and the judgment made. section 8.3 section 8.6
Demand and churn prediction
A bike shop needs churn and demand predictions across locales and product types. Its selection requirements include confidence intervals and parameters for what-if scenarios. Expected usage makes inference cost and throughput less pressing, while ordinary web-app latency remains relevant. These requirements shape which modeling approaches and results are useful; they do not, by themselves, establish a winning model. section 8.5
Plan the effort to match the risk. Some projects can rely on offline tests; others need online testing, while qualitative selection or calibration may be unnecessary. Keep tests and decisions documented and reproducible, and agree with stakeholders when data or technical risks require a change to the project roadmap. section 8.1 section 4.7
