Chapter 2: Meet your data!

Introduction

The chapter introduction distinguishes the data typically used for statistical analysis from unstructured data and explains how simplified one-variable examples relate to real-world tables.

• Structured and unstructured data: Explain why statistical analysis often uses structured, tabular data, while unstructured documents, audio, and images are impractical for manual statistical analysis.
  Why it matters: The form and complexity of data affect how practically it can be analyzed.
• Single variables and real tables: Distinguish the chapter’s single-variable example datasets from the multivariable tables usually encountered in real-world work.
  Why it matters: It helps you relate a simple example dataset to the wider tables you may encounter in practice.

Key terms: structured data, tabular data, unstructured data, perceptual models, single variable, multiple variables, columns

2.1 What is data?

Defines data as recorded measurements or observations and explains why understanding it matters before building statistical models.

• Definition of data: State that data consists of recorded measurements or observations collected in a hopefully systematic way, rather than merely calling it information.
  Why it matters: A concrete definition makes it easier to discuss what a dataset actually represents.
• Data as statistical foundation: Explain why the section asks readers to think carefully about data before doing statistics: statistical models depend on the data beneath them.
  Why it matters: Statistical analysis cannot make up for a weak grasp of the data it uses.

Key terms: data, recorded measurements, observations, data foundation, statistical models

2.2 Family photo experiment

The family-photo thought experiment shows why data offers incomplete, potentially biased clues and why analysis should begin with a focused question while still examining how the data was produced.

• Limits of data snapshots: Explain why a photograph, like any collected data, cannot reveal a complete story: it captures particular moments and leaves out context, including events outside the frame or between observations.
  Why it matters: Remembering what data leaves out helps you avoid turning a few observations into an unwarranted story.
• Question first data collection: Justify starting with a specific question and collecting data for it rather than gathering data without an objective and searching for findings afterward. Use the father’s golf ability as the section’s example of a narrower, more workable question.
  Why it matters: A clear question guides what evidence to collect and makes overreaching conclusions less likely.
• Data provenance and interpretation: Explain why a focused question does not eliminate the need to ask how data was created, what context is missing, and whether the observations reliably represent the domain of interest.
  Why it matters: Checking where evidence came from can prevent a plausible-looking observation from supporting the wrong conclusion.

Key terms: data, snapshots, the Instagram problem, objective, scientific method, data mining, spurious findings, context, biases, representative

2.3 Where does data come from?

The section identifies four common ways data is created and explains why its origin matters when judging whether it suits an analysis.

• Four common data sources: Distinguish sensors, software activity, manual human effort, and synthetic data by how each creates data, giving an example of each.
  Why it matters: Recognizing how data was made helps you decide what questions to ask about it.
• Limits of sensor measurements: Explain why sensor data captures only part of what it is intended to describe, using the camera or weather-station example.
  Why it matters: A sensor reading is not automatically a complete picture of the environment.
• Source provenance and suitability: Explain why an analyst should ask how data was created, including when it was bought from a third party, and check what is missing, how current it is, and whether it reflects the problem.
  Why it matters: Knowing where data came from is a starting point, not a guarantee that it answers your question.

Key terms: sensors, software activity, manual human effort, synthetic data, calibration, sampling point, data source, third party, missing, updated

2.4 Samples and populations

Introduces the idea that observed data is often a sample of a larger population it is meant to represent.

• Data as sample and population: Explain why it is often useful to treat the data at hand as a sample of a larger population, and why this distinction matters beyond controlled studies.
  Why it matters: This distinction helps you think about what the data in front of you is meant to represent.

Key terms: sample, population

2.5 What is a population?

Defines a population by its boundaries and explains why large or theoretical populations usually must be studied through samples.

• Population boundaries and membership: Explain that a population is the group of interest in a study, defined by boundaries that distinguish what belongs from what does not, even when those boundaries are fuzzy.
  Why it matters: A clear population tells you whom or what a measurement is meant to describe.
• Census and sampling limits: Explain why researchers often use sampling to estimate measurements for a population rather than collecting data from every member.
  Why it matters: It explains why a study's data often covers only part of the group it aims to describe.
• Theoretical populations and uncertainty: Describe how a population can consist of possible events rather than a fixed, accessible group, and why collected observations remain a sample of that population.
  Why it matters: It keeps observed data from being mistaken for every situation that could occur.

Key terms: population, boundaries, census, sampling, theoretical, edge cases, sample

2.6 What is a sample?

Defines a sample as data used to infer something about a larger population, then shows why sample size, selection, context, and the definition of the population matter.

• Samples and population inference: Explain what a sample is and how measurements from it can be used to infer an attribute of a population. Use the dorm sleep study to distinguish the 64-person population from the six-person sample and its estimated average.
  Why it matters: It keeps you from treating a result measured on a few people as a known fact about everyone.
• Conditions for representative sampling: Describe why random selection, sample size, and the context of measurement affect how well a sample represents its population.
  Why it matters: Even a measured average can be misleading if the people or circumstances measured do not reflect the wider group.
• Data as a sample: Explain why even a complete collection of observed data may be better treated as a sample, and describe the need to define the larger population it is meant to represent.
  Why it matters: Defining what your observed data stands for helps prevent claims that extend beyond what it can represent.

Key terms: sample, population, N, random, unbiased, representative, sample size, context, theoretical population, all possible road events

2.7 Sampling bias

This section explains how unrepresentative samples distort conclusions, using a student sleep study to show sources of sampling bias and ways to manage it.

• Sampling bias and population claims: Explain why an estimate from one dormitory cannot automatically describe students on a campus, across a country, or worldwide.
  Why it matters: A precise result for one group may still be a misleading claim about a broader group.
• Random selection and participation: Describe why sampling across the population at random helps with representativeness and why allowing selected students to opt in or out can still skew a sleep study.
  Why it matters: How people enter a study can change its result, even after the target population is chosen correctly.
• Recruitment and device-related skew: Explain how recruiting through social media and requiring sleep-tracking watches could each favor different kinds of respondents and distort the reported sleep average.
  Why it matters: Recruitment and eligibility choices can push an estimate in different directions.
• Managing unavoidable sampling bias: Describe the section’s practical response to unavoidable bias, including disclosure, random sampling, and deliberate demographic sampling, while recognizing that corrections involve judgment.
  Why it matters: A useful analysis examines and communicates its sampling limitations rather than treating the data as automatically representative.

Key terms: bias in data, sample, population, representative, unbiased sample, randomly select, self-selection bias, social media, sleep-tracking watches, bias

2.8 Numeric systems and data types

Introduces common data types and explains why recognizing what values represent matters when working with data.

• Numeric values and their meanings: Distinguish numbers used as counts or identifiers from numbers used to encode category labels or Boolean values. Explain that a numeric-looking value does not by itself reveal what the variable represents.
  Why it matters: Recognizing what a number stands for helps you interpret the data rather than relying on its appearance alone.
• Nonnumeric data for grouping: Describe the nonnumeric data types named here and how they can help organize or group data.
  Why it matters: Not every useful variable is numeric; nonnumeric values can help organize the measures you examine.

Key terms: natural numbers, integers, unique identifiers, counts, labels, Booleans, nonnumeric types, dates, strings, categories

2.9 Natural numbers

This section introduces natural numbers as counting values and explains how the decimal system writes large numbers using columns of digits.

• Natural numbers in count data: Explain why counts in data are discrete whole-number values, and distinguish positive natural numbers from the debated inclusion of zero.
  Why it matters: It helps you recognize which data values represent counts.
• Decimal digits and columns: Describe how the decimal system uses the digits 0 through 9 in columns to express numbers beyond nine, and distinguish that system from floating-point values.
  Why it matters: It separates how a number is written from whether that number has a fractional part.

Key terms: natural numbers, whole numbers, discrete, decimal system, digits, column system, floating-point values

2.10 Integers

Defines integers and shows why negative whole numbers are useful for describing real quantities.

• Integer number set: Describe integers as positive and negative whole numbers, including zero, and explain how they relate to natural numbers.
  Why it matters: This lets you recognize which values in data are integers.
• Negative quantities in context: Explain how a negative integer can represent a quantity below zero, using an example from finance, temperature, or elevation.
  Why it matters: A negative value can describe a meaningful loss or a position below a reference point.

Key terms: integers, natural numbers, whole numbers, negative numbers, loss, sea level

2.11 Real numbers

This section introduces real numbers, distinguishes rational from irrational numbers, and explains why computer floating-point values are approximations.

• Real numbers for partial measurements: Explain why real numbers are useful for measurements that fall between whole-number values, such as a half gallon of milk or 5.32 miles.
  Why it matters: It explains why measured data often needs values between whole numbers.
• Rational and irrational real numbers: Distinguish rational real numbers, which can be expressed as ratios, from irrational real numbers, which cannot.
  Why it matters: The distinction shows that real numbers include both values expressible as ratios and values that are not.
• Floating-point approximation limits: Explain why Python floating-point values approximate numbers and can produce small precision errors, and describe the section’s example of avoiding underflow.
  Why it matters: It prevents a reader from treating a computer’s displayed decimal as an unlimited-precision value.

Key terms: real numbers, floating-point decimal, integers, rational number, irrational numbers, ratio, floating-point value, floating-point precision errors, floating-point underflow, logarithmic addition

2.12 Mean, median, percentile, and mode

Introduces measures of central tendency and descriptive statistics as ways to summarize numerical data.

• Different meanings of center: Explain that measures of central tendency identify a typical or central value in a numerical dataset, but that different measures define the center differently and have different strengths and weaknesses. Describe the mode as identifying the most frequent value, particularly for discrete values such as integers or counts.
  Why it matters: Choosing a summary starts with recognizing that different measures answer different questions about what is typical.
• Numerical data summaries: Explain how descriptive statistics summarize numerical variables by condensing large datasets into a few key numbers, such as the mean, median, or percentiles.
  Why it matters: A few summary numbers can make a large numerical dataset manageable to inspect.

Key terms: measures of central tendency, mean, median, mode, discrete values, numerical variables, descriptive statistics, percentiles

2.13 Mean

This section explains how the mean represents the center of numerical data, how its notation differs for samples and populations, and how weights change each value’s influence.

• Mean as numerical center: Explain how to calculate the mean and what it represents for a set of numerical values.
  Why it matters: The mean gives you a single value that locates the center of a numerical dataset.
• Sample and population mean notation: Distinguish the notation for sample and population means, and explain what the summation symbol means.
  Why it matters: The symbols let you tell whether a reported mean describes a sample or an entire population.
• Weighted mean and influence: Explain how a weighted mean differs from an ordinary mean and how weights affect the result.
  Why it matters: Weighting lets an average reflect a deliberate difference in how much each value should count.

Key terms: mean, average, central location, x-bar, μ, n, Σxᵢ, weighted mean, weighted average, weight

2.14 Median

Defines the median, shows how to find it for odd and even numbers of values, and explains why it can better represent a typical value when outliers skew the mean.

• Finding the median: Describe how to find the median by ordering the values and selecting the middle one, or averaging the two middle values when the count is even.
  Why it matters: Finding the middle correctly lets you describe a dataset without relying on its arithmetic mean.
• Median under extreme values: Explain why the median can better represent a typical value than the mean when an outlier is present, using the bar-tab example.
  Why it matters: This distinction helps you choose a number that reflects a typical experience rather than an exceptional one.

Key terms: median, ordered sequence, centermost values, mean, outlier, skewed

2.15 Percentile

Defines percentiles and quartiles, uses the interquartile range to describe the middle half of a dataset, and notes how quantiles relate to percentiles.

• Percentiles and quartile positions: Explain what a percentile means in ordered data and how the 25th, 50th, and 75th percentiles divide the data into quarters.
  Why it matters: Percentiles let you describe where values sit in an ordered dataset.
• Interquartile range and spread: Describe what the interquartile range measures and calculate it from the bar-tab quartiles in the section.
  Why it matters: The interquartile range adds a measure of spread to the median without focusing on extreme values.
• Quantiles and percentile estimation: State how quantiles relate to percentiles and why a percentile from a small dataset may be interpolated.
  Why it matters: This helps you recognize equivalent percentile and quantile labels without assuming every reported value is an observed data point.

Key terms: percentile, median, quartiles, interquartile range, 25th percentile, 75th percentile, outliers, quantile, interpolation, numpy

2.16 Mode

This section defines the mode, shows how it identifies the most common booking-party size, and explains when data have no mode or are called bimodal.

• Mode and its uses: Explain that the mode is the value or values occurring most often, and describe why it is useful for categorical or repetitive data. Apply this to the booking example, where party size 2 occurs five times.
  Why it matters: The mode gives you a way to describe what occurs most often, even when an average is not useful.
• Absent and multiple modes: Distinguish data with no mode from data with two modes, and describe the section’s additional use of “bimodal” for data clustered around multiple values.
  Why it matters: These cases prevent you from reporting a single most common value when the data do not have one.

Key terms: mode, categorical, discrete types of data, bimodal, multiple local maxima

2.17 Variance and standard deviation

The section explains why a central value alone can be misleading and introduces variance and standard deviation as measures of data spread.

• Dispersion and reliable decisions: Explain that dispersion is how much data varies around a central value, and why substantial variation can make predictions and decisions less reliable even when the mean is known.
  Why it matters: Knowing an average is not enough if individual cases may differ greatly from it.
• Variance and standard deviation: Describe variance and standard deviation as measures of how much dataset values deviate from the mean, using squared differences between each value and the mean.
  Why it matters: These measures put a number on how tightly or loosely values cluster around the mean.

Key terms: dispersion, mean, median, variance, standard deviation, squared differences

2.18 Variance

This section explains how variance measures spread, why sample variance uses n − 1, and how to calculate and interpret it in a temperature example.

• Variance as data spread: Explain how variance is calculated and what high or low variance says about values around their mean. Describe why spread matters when judging the reliability of model predictions.
  Why it matters: A mean alone does not tell you how consistently the data—or predictions based on it—behaves.
• Sample variance correction: Distinguish the divisors used for sample and population variance, and explain why sample variance uses n − 1.
  Why it matters: Using the right divisor keeps a sample calculation from being treated like a calculation over the whole population.
• Temperature variance calculation: Apply the section’s sample-variance method to the temperature example, including the NumPy setting needed to match the sample calculation.
  Why it matters: It connects the idea of sample variance to a calculation a reader could reproduce in code.
• Square root counterpart: Explain why variance can be hard to interpret on its own and how standard deviation relates to it.
  Why it matters: This explains why the next measure starts with the variance but is easier to relate to the original data.

Key terms: variance, mean, variability, sample variance, population variance, Bessel’s correction, n − 1, len(temps) - 1, var(), ddof=1

2.19 Standard deviation

This section defines standard deviation as the square root of variance, explains why it is easier to interpret, and notes how to calculate it for a sample and when outliers may make it less useful.

• Standard deviation and variance: Explain that standard deviation is the square root of variance and describes spread in the same units as the original measurements.
  Why it matters: Standard deviation lets you discuss variation in units that make sense for the measurements.
• Sample standard deviation in NumPy: Describe how the section's standard_deviation() function obtains its result and how NumPy's std() is set to treat data as a sample.
  Why it matters: The sample setting matters when comparing a hand-built calculation with NumPy's result.
• Outlier-sensitive spread measures: Explain the section's warning about standard deviation when data has outliers and identify the alternative it suggests.
  Why it matters: A measure of spread can give a misleading impression when outliers distort it.

Key terms: standard deviation, variance, spread, variance(), standard_deviation(), numpy’s std(), ddof=1, sample, outliers, interquartile range

2.20 Real-world example with tornadoes

The section uses NOAA tornado data to show how distributions and grouped summaries can suggest a relationship between tornado width and severity while leaving room for uncertainty.

• Dataset fields and limitations: Describe what the NOAA tornado dataset measures for this investigation and why its documentation and collection limits matter.
  Why it matters: Knowing what the fields mean and how observations were collected keeps the analysis grounded in what the data can support.
• Histogram and skewed widths: Explain how the histogram and summary statistics reveal the tornado-width distribution and why the median and quartiles help describe it.
  Why it matters: The shape of the data determines which summaries give a useful account of tornado width.
• Severity-group width comparison: Describe how comparing median widths by EF rating supports a tentative relationship between width and severity.
  Why it matters: Comparing like-for-like groups turns a pattern in the dataset into a testable idea without overstating the evidence.
• EF4 spread and sample size: Explain why the smaller EF4 interquartile range may not establish that EF4 widths are consistently similar.
  Why it matters: A small group's apparently tight spread should not be mistaken for a reliable picture of all such tornadoes.

Key terms: NOAA, TOR_WIDTH, TOR_F_SCALE, EFU, histogram, bins, right skew, mean, median, interquartile range

2.21 Summary