Chapter 24: Exploring data

Introduction

The introduction previews a shift from getting and cleaning data to manipulating and exploring it with Python.

24.1 Python tools for data exploration

The section introduces pandas and Matplotlib as tools the chapter will briefly explore for working with data in Jupyter.

24.1.1 Python’s advantages for exploring data

This subsection explains why Python is useful for exploring data despite not always having the fastest raw performance.

• Python data exploration advantages: Explain why Python can be a strong choice for manipulating and exploring data even when its raw performance is not the fastest.
  Why it matters: It helps explain why raw execution speed is only one factor when choosing a tool for data exploration.

Key terms: raw performance, NumPy, readability, accessibility, developer time

24.1.2 Python can be better than a spreadsheet

This section explains why Python can be more flexible than spreadsheets for manipulating large or complex datasets.

• Spreadsheet limits and Python flexibility: Explain why Python can be a better choice than a spreadsheet for some data manipulation, while acknowledging what spreadsheets do well.
  Why it matters: It helps you recognize when a spreadsheet’s size or structure may constrain the way you work with data.

Key terms: row limit, two-dimensional grids, lists, tuples, sets, dictionaries

24.2 Python and pandas

This section explains why pandas is useful for repetitive data-handling work in Python.

• Pandas automation rationale: Explain why pandas was created for exploring and manipulating datasets, using the section’s examples of common operations.
  Why it matters: This gives you the reason for using pandas before you encounter its specific features.

Key terms: pandas, datasets, cleaning data, filtering data

24.2.1 Why you might want to use pandas

This section explains why pandas can make tabular data work easier and why its learning curve and fit for the task matter.

• Pandas benefits and tradeoffs: Explain why pandas is useful for manipulating and analyzing tabular or relational data, and why it may not be worth using for every task.
  Why it matters: This helps you decide whether learning pandas is worthwhile for the data work you need to do.

Key terms: pandas, tabular or relational data, standard framework

24.2.2 Installing pandas

This subsection explains how to get pandas and its plotting companion installed and how to prepare a Jupyter notebook to use them.

• Pandas installation by environment: Explain that Colaboratory already includes pandas, matplotlib, and numpy, while another environment can install pandas and matplotlib with pip from the command line of its Jupyter virtual environment.
  Why it matters: This helps you decide whether you need to install anything before exploring data.
• Jupyter plotting and import setup: Describe the purpose of the three suggested notebook lines: enable plots in the code cell, import pandas as pd, and import numpy. Explain that pandas relies on numpy, though the following examples do not use numpy explicitly.
  Why it matters: This setup makes the notebook examples more convenient to run and plot.

Key terms: Colaboratory, pandas, matplotlib, numpy, pip, Jupyter virtual environment, Jupyter “magic” function, pd

24.2.3 Data frames

This section introduces pandas data frames as in-memory grids and shows how their columns and values are easier to work with than a list of lists.

• Data frame structure and display: Describe a data frame as a two-dimensional, in-memory grid similar to a relational database table, and explain what changes when a list-of-lists grid is turned into one.
  Why it matters: A data frame makes tabular data easier to see and work with.
• Named column selection: Explain how column names let you select one column by name or select multiple columns as another data frame, and contrast this with extracting a column from a list of lists.
  Why it matters: Selecting by name is more convenient than remembering column positions and writing a comprehension.
• Frame-wide operations: Explain what the example’s add() method does to a data frame and why the section presents it as convenient.
  Why it matters: Frame-wide methods can express a change to all selected values without writing nested loops.

Key terms: data frame, list of lists, row, column, column names, select columns by name, double brackets, subset, list comprehension, add()

24.3 Data cleaning

The section introduces upcoming examples of cleaning data with pandas and comparing those operations with plain Python.

24.3.1 Loading and saving data with pandas

This section shows how pandas loads data into data frames, standardizes missing values while loading, and saves data frames in different formats.

• Pandas data loading options: Describe how pandas reads JSON and CSV files into data frames, and explain that some supported data sources require other libraries.
  Why it matters: It helps you choose a loading method and recognize when a loading error may come from a library pandas relies on.
• Missing value standardization: Explain how na_values can convert a file's markers for missing data to NaN during loading, including when a file uses several different markers.
  Why it matters: Consistent missing-value markers make the loaded data easier to work with.
• Data frame output formats: Describe that a pandas data frame offers methods for saving its contents in different forms, including writing a file or producing JSON.
  Why it matters: Loading data into a frame does not lock you into its original format when you need to save the result.

Key terms: read_json(), read_csv(), data frame, SQLAlchemy, na_values, NaN, Missing, JSON object

24.3.2 Data cleaning with a data frame

Shows how pandas data frames simplify cleaning a temperature CSV by loading only useful columns and converting a percentage column from text to numeric values.

• Useful columns on load: Explain how the example makes the temperature data easier to work with by assigning numeric column names and loading only the columns of interest.
  Why it matters: Leaving out unneeded fields makes the data frame easier to clean and explore.
• Percentage column conversion: Trace how the example cleans the heat-index coverage column, from strings ending in % to numeric values and optionally fractions.
  Why it matters: The example illustrates how column-wide operations streamline a common data-cleaning task.

Key terms: data frame, range(), names parameter, NaN, strip(), replace(), div(), chaining, percentage, fraction

24.4 Data aggregation and manipulation

Introduces the upcoming examples of merging, aggregating, grouping, and filtering data with pandas.

24.4.1 Merging data frames

This subsection shows how pandas merges related data frames and names the kinds of joins it supports.

• Merging related data frames: Explain how pandas merge links two data frames on specified columns to produce a new frame containing columns from both, such as sales-call activity and territory revenue.
  Why it matters: It lets you connect information kept in separate datasets for analysis.
• Supported join relationships: State that pandas merge is not limited to matching one row in each data frame: it can also perform one-to-many, many-to-many, right, and left joins.
  Why it matters: The example is only one of the ways merge can relate datasets.

Key terms: merge function, data frame, relational database join, one-to-one, one-to-many, many-to-many, right and left joins

24.4.2 Selecting data

This subsection shows how pandas selects data-frame rows by using a condition as an index.

• Conditional data-frame row selection: Explain how to select rows from a pandas data frame by using a condition as its index, such as selecting sales where the territory equals 3 or the amount per call exceeds 500.
  Why it matters: You can focus on the rows you need without writing a more cumbersome plain-Python filtering structure.

Key terms: data frame, select, filter, index

24.4.3 Grouping and aggregation

This section shows how to summarize pandas columns, select rows using a summary value, and aggregate data by groups.

• Column summaries and median selection: Explain how pandas column methods provide sums, means, medians, minimums, and maximums, and how a median can be used in a selection to find rows whose amount per call is above it.
  Why it matters: A summary can help you identify records that stand out from the rest.
• Grouped column aggregation: Describe how to select columns to aggregate, group rows by another column with a data frame's groupby method, and sum values within each group, such as calls and amounts by month or territory.
  Why it matters: Grouping turns individual records into useful totals for each month, territory, or other category.

Key terms: column methods, median, selection operation, groupby, aggregate, sum

24.5 Plotting data

Shows how to create simple plots directly from a pandas data frame in a Jupyter notebook.

• Data frame plotting methods: Explain how pandas uses matplotlib to plot data directly from a data frame in a Jupyter notebook, and describe the chart types produced by the methods shown.
  Why it matters: You can quickly visualize data you have already prepared in a data frame.

Key terms: pandas, matplotlib, data frame, inline plotting, .plot.bar(), .plot.line()

24.6 Why you might not want to use pandas

This section explains why pandas is useful but not always the best tool for a data task.

• Choosing pandas or plain Python: Explain why tool choice should depend on the problem rather than defaulting to pandas. Describe how the effort of learning pandas and the shape and needs of the data can favor plain Python.
  Why it matters: Choosing a tool that fits the data and task can save effort even when pandas is available.

Key terms: pandas, plain old Python, production situations, large datasets

24.7 Summary