Parallelism executes multiple computational operations concurrently instead of one after another. In outlier detection, it can reduce training or prediction time when hardware, detector independence, and model support allow it. section 10.2.4
Apply parallelism in several ways:
- Set a detector's n_jobs parameter to control how many processes run simultaneously; PyOD supports this for COPOD, ECOD, Feature Bagging, and Isolation Forest. section 10.2.4
- Train independent detectors concurrently with Python multiprocessing or SUOD, especially when separate models are created for manually segmented data. section 10.2.4 section 10.2.3
- Predict concurrently across detectors or split a large prediction batch into smaller batches when train and predict steps are separate. section 10.2.4
- Do not assume concurrency is fastest: sequential filtering can reduce later workloads when only records flagged by every detector matter. section 10.2.4
Listing 10.5 Executing prediction in parallel
import numpy as np
import pandas as pd
from sklearn.datasets import fetch_kddcup99
from sklearn.preprocessing import OrdinalEncoder
from pyod.models.iforest import IForest
import concurrent
import warnings
warnings.filterwarnings('ignore')
np.random.seed(0)
X, y = fetch_kddcup99(subset="SA", percent10=True, random_state=42,
return_X_y=True, as_frame=True)
enc = OrdinalEncoder() #1
X = pd.DataFrame(enc.fit_transform(X), columns=X.columns)
det = IForest()
det.fit(X)
num_splits = 5 #2
def evaluate_subset(df): #3
return det.decision_function(df)
process_arr = []
full_results = []
rows_per_subset = len(X) // num_splits
with concurrent.futures.ProcessPoolExecutor() as executor: #4
for dataset_idx in range(num_splits):
print(f"Starting process for dataset: {dataset_idx}")
start_row = dataset_idx * rows_per_subset
end_row = (dataset_idx + 1) * rows_per_subset
print(start_row, end_row)
f = executor.submit(evaluate_subset, X[start_row: end_row+1])
process_arr.append(f)
for f in process_arr: #5
full_results.extend(f.result())
A Python example divides KDD Cup records into five subsets and evaluates them with concurrent processes using a PyOD Isolation Forest. section 10.2.4
Sequential detector filtering
A KDD Cup workflow can apply Isolation Forest to the full dataset, retain flagged records, then apply LOF and GMM to progressively smaller subsets. This may outperform parallel execution when later detectors process substantially fewer records. section 10.2.4
