Spark is an open-source distributed computing engine for processing and analyzing large datasets. PySpark brings Spark workflows to Python, using dataframes similar to pandas. section 10.2.5
Use Spark in outlier-detection workflows to:
- Run univariate tests based on z-scores, interquartile ranges, or median absolute deviation directly on Spark dataframes. section 10.2.5
- Filter strong outliers before exporting data to NumPy for other detection algorithms. section 10.2.5
- Scale parallel model execution with projects such as spark-iforest, which provides Isolation Forest for large datasets. section 10.2.5
Large-volume outlier preprocessing
Load data into Spark dataframes, apply a simple univariate outlier test, remove strong outliers, and export the filtered data to NumPy for additional detectors. section 10.2.5
