A data frame is an in-memory table used to process columns of tabular data. In pandas, its usefulness depends not just on what the table contains, but on how columns are loaded, typed, and accessed. A large CSV can become a much larger data frame when type inference selects expensive representations. section 7.0 section 7.1.2
Measure the table before changing it
The January 2020 New York City taxi CSV expands from a 566 MB file to a 2 GB data frame. Run `df.info(memory_usage="deep")`, then inspect each column’s type, number of unique values, and deep memory usage. In this case, date-time strings stored as `object` take about 464 MB per column. Converting the two date-time columns with `pd.to_datetime` cuts each to about 48 MB and the whole frame to 1.2 GB. A mixed-type warning is a reason to inspect columns, not simply to set `low_memory=False`: that setting can exhaust memory on large inputs. section 7.1 section 7.1.2
Shape the data frame around the computation:
- Load only needed columns with `usecols`; specify suitable types with `dtype` and date-time columns with `parse_dates`. Check missing values before narrowing a numeric type: the taxi example fills missing payment types before converting them to NumPy `int8`. section 7.1.2 section 7.2
- Benchmark access patterns rather than treating them as interchangeable. For the taxi-tip calculation, explicit `iloc` loops and `iterrows` are slow; `itertuples` improves the loop substantially, while column-wise vectorization avoids row traversal. Also check the result being computed: a ratio of column sums is not the same calculation as the mean of per-row ratios. section 7.2.2
- When the whole table cannot fit in memory, pass `chunksize` to `pd.read_csv`. It returns an iterator whose chunks are data frames, rather than one complete data frame. Keep column types consistent across chunks when converting them to Arrow tables for a Parquet file. section 8.3.2
A data frame is also a boundary between processing and storage formats. Apache Arrow can load data for pandas and convert pandas frames to Arrow tables; Parquet stores heterogeneous columnar data so applications can access needed columns. These alternatives are worth measuring against direct pandas loading, rather than assuming that conversion is free or always faster. section 7.0 section 8.2
