Dask is a Python library for parallel and distributed computing with partitioned data that exceeds available memory. Its pandas-like API supports collecting and preprocessing large datasets, while basic univariate outlier tests can run across distributed data. section 10.2.5
A practical Dask outlier workflow can:
- Encode categorical columns, create a Dask dataframe from pandas data, and divide it into partitions. section 10.2.5
- Apply RobustScaler, compute each row’s maximum scaled value, and inspect the resulting records. section 10.2.5
- Use Dask mainly for collection and preprocessing when sophisticated outlier detection must be performed elsewhere. section 10.2.5
Listing 10.8 Identifying univariate outliers in Dask
import pandas as pd
import numpy as np
import dask.dataframe as dd
from sklearn.preprocessing import RobustScaler, OrdinalEncoder
from sklearn.datasets import fetch_kddcup99
X, y = fetch_kddcup99(subset="SA", percent10=True, random_state=42,
return_X_y=True, as_frame=True)
enc = OrdinalEncoder() #1
df = pd.DataFrame(enc.fit_transform(X), columns=X.columns)
dask_df = dd.from_pandas(df, npartitions=2) #2
transformer = RobustScaler().fit(dask_df) #3
vals = transformer.transform(dask_df)
dask_df = dd.from_array(vals, columns=df.columns)
dask_df['Max Value'] = dask_df.max(axis=1) #4
dask_df.compute().sort_values(['Max Value']) #5
Runnable example showing encoding, partitioning, robust scaling, univariate scoring, computation, and sorting. section 10.2.5
