liveBook · Manning

Read_csv

As explained in Statistics Every Programmer Needs

read_csv is a pandas function that reads comma-separated or otherwise delimited text data into a data frame.

`pd.read_csv()` loads a CSV file into a pandas data frame so its columns can be inspected, cleaned, and analyzed. Import pandas first, then pass a quoted filename: `import pandas as pd; mds = pd.read_csv('mds.csv')`. A filename alone works when the file is in the working directory; otherwise, pass a relative or full path. By default, the first row supplies column headers. section 4.2.1

Check the result before analysis. `mds.info()` reports row count, columns, non-null counts, and inferred data types; `mds.describe()` summarizes its numeric columns. For a small data frame, printing every row is practical. For a larger one, inspect selected rows with `head()` and `tail()`. Reading a file creates the data frame, but does not establish that its values or assumptions are suitable for a model. section 4.2.1 section 5.2.1

Limit the import when only part of a file is needed:

  • Pass column names to `usecols`, as in `pd.read_csv('nfl_pbp.csv', usecols=['QUARTER', 'DOWN', 'TO_GO'])`. Selecting relevant play-by-play variables at load time reduces memory use and speeds loading compared with importing the entire large data set and then subsetting it. section 6.2
  • Pass zero-based column positions when those are more convenient: `pd.read_csv('street_address_listing.csv', usecols=[0])` imports only the first column; `usecols=[1]` imports the second column of the population file. section 12.4.1 section 12.4.2
>>> import pandas as pd
>>> nfl_pbp = pd.read_csv('nfl_pbp.csv',
>>>                    usecols = ['QUARTER', 
>>>                               'DOWN', 
>>>                               'TO_GO', 
>>>                               'OFFENSIVE_TEAM_VENUE',
>>>                               'SCORE_DIFFERENTIAL', 
>>>                               'PLAY_TYPE', 
>>>                               'YARDS_GAINED'])

The football import shows a practical `usecols` call: load only the variables needed to predict fourth-down conversion. section 6.2

Inspect before extracting leading digits

Loading `street_address_listing.csv` with `usecols=[0]` produces an `ADDRESS_NO` column with 194,836 non-null entries. Its reported `object` type matters: addresses can contain text such as suite information. Extract the first character, inspect the distinct results, discard non-digit initials, and only then convert the retained leading digits to integers for Benford analysis. A complete non-null count alone would not catch those non-digit values. section 12.4.1

How Statistics Every Programmer Needs covers it

How deep it goes
Demonstrates filename and path handling, default headers, `usecols`, and post-import inspection; does not develop other parser options.
Examples
Imports marathon split times, raisin measurements, football plays, street addresses, and population counts; previews imported rows and metadata.
What it assumes
Requires a CSV file and a pandas import; explains working-directory filenames, alternative paths, and the default header assumption.
What you can do after
Apply it via pandas to load a CSV, select needed columns, and inspect the resulting data frame before analysis.
How it connects
Connects pandas data frames and `usecols` to data exploration, data cleaning, logistic regression, and Benford’s law.

Also explained in

Compare how 5 books explain read_csv →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Statistics Every Programmer Needs. The section links point to the parts of the book it draws on.