Parquet is a compressed columnar file format for storing intermediate analytical datasets. It preserves type information, so date columns remain typed when the data is read again. section 4.4.1
Use Parquet to separate data cleaning from analysis: export cleaned data once, then reload it without repeating potentially slow preparation steps. section 4.4.1
Export a cleaned dataset
After adding corrected categories, brands, and product names, write the intermediate dataset with pandas: events.to_parquet("./data/events.parquet.gz", compression="gzip"). section 4.4.1
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
november = pd.read_csv("./data/november.csv.gz")
december = pd.read_csv("./data/december.csv.gz")
events = pd.concat([november, december], axis=0, ignore_index=True)
print(events.shape)
Inspect the event data that can be cleaned and exported as an intermediate Parquet dataset. section 4.4.1
