Memory usage measures how much computer memory a pandas DataFrame or computation occupies. Large datasets make reducing that footprint a practical performance task. section 12.13
Keep only the taxi columns you need
A New York City taxi CSV contains 7,667,792 rides. Loading only the required columns with pd.read_csv(usecols=...) reduced memory usage from 2.4 GB to 234 MB, leaving a four-column DataFrame for analysis. section 3.2
Practical ways to reduce resource use include:
- Choose columns carefully instead of retaining unused data. section 12.13
- Use categorical representations where appropriate. section 12.13
- Read Feather rather than CSV when the workflow permits. section 12.13
- Specify columns or manage dtype inference when importing wide data. section 3.2 section 8.2
