1 Optimizing systems by experiment

The past 20 years have seen a surge in interest in the development of experimental methods used to measure and improve engineered systems, such as web products, automated trading systems, and software infrastructure. Experimental methods have become more automated and more efficient. They have scaled up to large systems like search engines or social media sites. These methods generate continuous, automated performance improvement of live production systems.

Using these experimental methods, engineers measure the business impact of the changes they make to their systems and determine the optimal settings under which to run them. We call this process experimental optimization.

This book teaches several experimental optimization methods used by engineers working in trading and technology. We’ll discuss systems built by three specific types of engineers:

Machine learning engineers often work on web products like search engines, recommender systems, and ad placement systems. Quants build automated trading systems. Software engineers build infrastructure and tooling such as web servers, compilers, and event processing systems.

1.1 Examples of engineering workflows

1.1.1 Machine learning engineer’s workflow

1.1.2 Quantitative trader’s workflow

1.1.3 Software engineer’s workflow

1.2 Measuring by experiment

1.2.1 Experimental methods

1.2.2 Practical problems and pitfalls

1.3 Why are experiments necessary?

1.3.1 Domain knowledge

1.3.2 Offline model quality

1.3.3 Simulation