4 Database trouble and testing in production


This chapter covers

  • Designing chaos experiments for open source software
  • Adding network latency by using Traffic Control
  • Understanding when testing in production might make sense and how to approach it

In this chapter, you will apply everything you’ve learned about chaos engineering so far in a real-world example of a common application you might be familiar with. Have you heard of WordPress? It’s a popular blogging engine and content management system. According to some estimates, WordPress accounts for more than a third of all pages on the internet, and for most CMS-backed websites (http://mng.bz/e58Q). It’s typically paired with a MySQL database, another popular piece of technology.

Let’s take a vanilla instance of WordPress backed by MySQL and, using chaos engineering, try to gain confidence in how reliably you can run it. You’ll try to preemptively guess what conditions might disturb it and design experiments to verify how it fares. Ready? Let’s see what our friends from Glanden are up to these days.

4.1 We’re doing WordPress

4.2 Weak links

4.2.1 Experiment 1: Slow disks

4.2.2 Experiment 2: Slow connection

4.3 Testing in production