Simulation estimates a system’s business metrics offline by combining logged data with user or market models, scientific models, or heuristics. Unlike simple fitting metrics such as cross-entropy or R2, it represents the system’s components and produces outcomes such as revenue, engagement, profit, or CPU time. section 1.3.3
Use simulation as a fast, inexpensive supplement to experimentation. It can process months or years of data in minutes or hours and rerun many strategy variants, but its results depend on modeled behavior and may be biased by missing counterfactuals. Experiments remain the more accurate measurement of business impact. section 1.3 section 1.3.3
A practical simulation workflow:
- Feed historical or production-log data into the same strategy or system logic used in production. section 1.3.3 section 6.1.1
- Model or approximate the environment’s response when the system takes an action. section 1.3.3
- Return a business or operational metric, then compare variants offline before selecting candidates for production evaluation. section 1.3.3 section 1.4
Trading and compiler simulations
A trading simulator replays historical trades and quotes through production strategy code, mimics market behavior, and estimates profit, risk, and shares traded. A compiler simulator configures JIT parameters, compiles a web server, replays production requests, and returns average CPU time with simulated measurement noise. section 1.3.3 section 6.1.1
Listing 6.1 JIT CPU time simulator
def jit_plus_server(parameters):
x = np.array(parameters) #1
d = len(x)
x1 = x - 0.15*np.ones(shape=(d,)) #2
x2 = x - 0.85*np.ones(shape=(d,)) #2
cpu_time = 2 - np.exp(-10*x1**2) - 0.5*np.exp(-10*x2**2) #2
return cpu_time.mean() + .005*np.random.normal() #3
Inspect a compact Python simulator that maps JIT parameters to CPU time and adds random variation, making the system suitable for black-box optimization. section 6.1.1
