A data frame is a tabular data structure with labeled rows and columns, suited to storing and manipulating heterogeneous data. It provides a common working surface for data preparation, scaling, filtering, and analysis. section 10.2.5 section 17.3.3
Pandas data frames work well when the dataset fits available resources. For larger datasets, Spark and Dask provide similar dataframe-oriented workflows, including preprocessing and basic univariate outlier tests. section 10.2.5
Typical uses include:
- Representing model input with named columns and indexed records. section 17.3.3
- Applying transformations such as encoding and robust scaling before calculating row-level values. section 10.2.5
- Filtering or ranking records, including records with unusually large values. section 10.2.5
Time-series input
A forecasting workflow can store timestamps in a column named ds and tracked values in a column named y. Forecast results return another data frame containing predictions and confidence bounds, which can be compared with the observed values. section 17.3.3
Listing 10.8 Identifying univariate outliers in Dask
import pandas as pd
import numpy as np
import dask.dataframe as dd
from sklearn.preprocessing import RobustScaler, OrdinalEncoder
from sklearn.datasets import fetch_kddcup99
X, y = fetch_kddcup99(subset="SA", percent10=True, random_state=42,
return_X_y=True, as_frame=True)
enc = OrdinalEncoder() #1
df = pd.DataFrame(enc.fit_transform(X), columns=X.columns)
dask_df = dd.from_pandas(df, npartitions=2) #2
transformer = RobustScaler().fit(dask_df) #3
vals = transformer.transform(dask_df)
dask_df = dd.from_array(vals, columns=df.columns)
dask_df['Max Value'] = dask_df.max(axis=1) #4
dask_df.compute().sort_values(['Max Value']) #5
The example converts encoded pandas data into a two-partition Dask data frame, applies RobustScaler, computes each row's maximum value, and sorts the computed results. section 10.2.5
