7 Statistical hypothesis testing

 

This section covers

  • Comparing sample means to population means
  • Comparing means of two distinct samples
  • What is statistical significance?
  • Common statistical errors and how to avoid them

Many ordinary people are forced to make hard choices every day. This is especially true of jurors in the American justice system. Jurors preside over a defendant’s fate during a trial. They consider the evidence and then decide between two competing hypotheses:

  • The defendant is innocent.
  • The defendant is guilty.

The two hypotheses are not weighted equally: the defendant is presumed to be innocent until proven guilty. Thus, the jurors assume that the innocence hypothesis is true. They can only reject the innocence hypothesis if the prosecution’s evidence is convincing. Yet the evidence is rarely 100% conclusive, and some doubt of the defendant’s guilt remains. That doubt is factored into the legal process. The jury is instructed to accept the innocence hypothesis if there is “reasonable doubt” of the defendant’s guilt. They can only reject the innocence hypothesis if the defendant appears guilty “beyond a reasonable doubt.”

Reasonable doubt is an abstract concept that’s hard to define precisely. Nonetheless, we can distinguish between reasonable and unreasonable doubt across a range of real-world scenarios. Consider the following two trial cases:

7.1 Assessing the divergence between sample mean and population mean

7.2 Data dredging: Coming to false conclusions through oversampling

7.3 Bootstrapping with replacement: Testing a hypothesis when the population variance is unknown

7.4 Permutation testing: Comparing means of samples when the population parameters are unknown

Summary