A test harness is the test or mock application and supporting setup used to run candidate models and collect results in a controlled test environment. It works alongside the data pipeline, model artifacts, and feedback collection facilities. section 8.2.2
Treat a test run as a repeatable deployment task:
- Secure access and deploy the pipeline, harness, candidate models, and collection components. section 8.2.2
- Run smoke tests, load the required data and other components, execute the tests, and gather results. Script the sequence where practical: a missed check or out-of-order step can compromise the test. section 8.2.2
Match the harness to the decision being made. Latency and throughput figures are accurate only when test conditions mirror production; sustained tests on realistic hardware are needed to establish expected performance. An inventory is more straightforward for assessing model memory footprint and component cost. section 8.2.7
Demand prediction across countries
A test harness runs candidate models against sales-period and country-group data sets. Similar overall scores conceal weaker performance in small countries, prompting a country-to-model mapping. Because investigation has exposed test information to the team, fresh large- and small-country data sets are used to retest the composite model. section 8.5
Before production release, put the harness under a signed test plan and complete testing alongside the operational handover and release work. section 9.10
