concept disk in category chaos engineering

appears as: disk, disk, disks, disks
Chaos Engineering

This is an excerpt from Manning's book Chaos Engineering.

In this chapter, we’re going to talk a lot about utilization and saturation of resources. A resource is any of the physical components making up a physical server, like CPU, disk, networking devices and RAM. There can also be software resources, like threads, pids, or inode ids. Utilization of a resource is an average time or proportion of the resource used. For example, for a CPU, a meaningful metric will be the percentage of time spent doing work. For a disk, the percentage of the disk that is full can be a meaningful metric, but so can its throughput. Finally, saturation is the amount of work that the resource can’t service at any given moment (often queued). Note, that a high saturation might be a sign of a problem, but it also may very well be desirable (for example in a batch processing system, where we want to use as close to 100% of the available processing power as possible).

#! /bin/bash
 
export NEW_FILESYSTEM_ROOT=${1:-~/new_filesystem}
export TOOLS="bash ls pwd mkdir ps touch rm cat vim mount" #A
 
echo "Step 1. Create a new folder for our new root"
mkdir $NEW_FILESYSTEM_ROOT
 
echo "Step 2. Copy some (very) minimal binaries"
for tool in $TOOLS; do
    cp -v --parents `which $tool` $NEW_FILESYSTEM_ROOT; #B
done
 
echo "Step 3. Copy over their libs"
# use ldd to find the dependencies of the tools we've just copied
echo -n > ~/.deps
for tool in $TOOLS; do
    ldd `which $tool` | egrep -o '(/usr)?/lib.*\.[0-9][0-9]?' >> ~/.deps #C
done
# copy them over to our new filesystem
cp -v --parents `cat ~/.deps | sort | uniq | xargs` $NEW_FILESYSTEM_ROOT #D
 
echo "Step 4. Home, sweet home"
NEW_HOME=$NEW_FILESYSTEM_ROOT/home/chaos
mkdir -p $NEW_HOME && echo $NEW_HOME created!
cat <<EOF > $NEW_HOME/.bashrc
echo "Welcome to the kind-of-container!"
EOF
 
echo "Done."
echo "To start, run: sudo chroot" $NEW_FILESYSTEM_ROOT #E

Ouch. A “mere” 100ms added latency to the MySQL database changes the average response time of Meower USA from 26ms to 490ms, or a factor of more than 18. If this sounds suspicious to you, that’s the reaction I’m hoping for!. To confirm our findings, let’s rerun the same experiment, but this time let’s use 1ms as the delay, the lowest that the tool will allow us. To add the delay, run the following command in the second terminal window:

pumba netem \
--duration 60s \
--tc-image gaiadocker/iproute2 \
delay \
--time 1 \ #A
--jitter 0 \
--correlation 0 \
"re2:meower_db"