A validation measurement checks whether a predicted optimum agrees with observed performance before a production system adopts the new setting. Run the system at the estimated optimum, aggregate many individual outcomes, and compare the observed result with the prediction. section 4.1.5
Apply the procedure as follows:
- Set the parameter or parameters to the values returned by optimization, such as threshold_opt and order_size_opt. section 4.1.5 section 4.2.2
- Collect enough individual production measurements to estimate an aggregate outcome and its standard error; the examples use 15,000 measurements. section 4.1.5 section 4.2.2
- Accept the estimate when it lies within the validation aggregate measurement plus or minus twice its standard error, an approximate 95% confidence interval. section 4.1.5 section 4.2.2
- Return to experimental design when the estimate falls outside those bounds or when measurements reveal an outlier that violates the local model assumption. section 4.2.2
Two-parameter trading experiment
Optimization suggests threshold_opt = 1.12, order_size_opt = 3.0, and estimated maximum markout profit = 0.41. A validation run produces an aggregate measurement of 0.37 with standard error 0.026, giving [0.32, 0.43]. Because 0.41 lies inside this interval, the estimate is validated. section 4.2.2
Figure 4.12 Validate stage. In this final stage, you’ll run the production prop trading system at the optimal value of threshold that you estimated in the analyze stage. If the measured markout profit at that threshold is close enough (see the next section for discussion) to the estimated markout_profit(threshold), then you may accept the optimal threshold as the result of the RSM process.
Use the validation-stage diagram to follow the handoff from an estimated optimum to a production run and an accept-or-redesign decision. section 4.1.5
