Accuracy measures how well detector predictions align with known outcomes. Assess it against labeled real data, synthetic test data, or pseudo-ground truth, using metrics suited to the task, such as AUROC or precision at k. section 14.2
Use accuracy as a practical comparison signal, not as a guarantee that a detector or ensemble is strong.
- Compare candidate ensembles with a chosen metric; the best combination may include detectors that are not individually top-performing. section 14.2
- Balance detector accuracy against diversity: minimizing correlation alone can favor weak detectors. section 14.4.5
- For rapid prediction workloads, consider trading some accuracy for speed, and estimate both under realistic workloads. section 8.7
Weighting detector scores
Doped records can serve as presumed outliers for evaluating detectors; their measured accuracy can then determine weights in a combined score. section 14.6.4
Listing 14.13 Creating an ensemble based on a weighted average
from sklearn.metrics import roc_auc_score
doped_df = df.sample(n=50) #1
for i in doped_df.index:
row = doped_df.loc[i]
col_name = np.random.choice(doped_df.columns)
doped_df.loc[i, col_name] = doped_df.loc[i, col_name] * -0.9
df['Doped'] = 0
doped_df['Doped'] = 1
full_df = pd.concat([df, doped_df])
full_df = full_df.reset_index(drop=True)
doped_df.head()
scores_df = pd.DataFrame()
np.random.seed(1)
det_if = IsolationForest() #2
det_if.fit(full_df)
scores_df['IF Scores'] = (-1)*det_if.score_samples(full_df)
det_lof = LocalOutlierFactor()
det_lof.fit(full_df)
scores_df['LOF Scores'] = (-1)*det_lof.negative_outlier_factor_
det_ee = EllipticEnvelope()
det_ee.fit(full_df)
scores_df['EE Scores'] = (-1)*det_ee.score_samples(full_df)
det_ocsvm = OneClassSVM()
det_ocsvm.fit(full_df)
scores_df['OCSVM Scores'] = (-1)*det_ocsvm.score_samples(full_df)
det_ns = NearestSampleOutlierDetector()
det_ns.fit(full_df)
scores_df['Nearest Samples Scores'] = det_ns.decision_function(full_df)
det_radius = RadiusOutlierDetector()
scores_df['Radius Scores'] = det_radius.fit_predict(full_df, radius=4.0)
total_accuracy = 0 #3
accuracy_dict = {}
for col_name in scores_df.columns:
auroc = roc_auc_score(full_df['Doped'], scores_df[col_name])
accuracy_dict[col_name] = auroc
total_accuracy += auroc
accuracy_dict = {x: accuracy_dict[x]/total_accuracy for x in accuracy_dict}
mod_scores_df = pd.DataFrame() #4
for col_name in scores_df:
scaler = RobustScaler()
mod_scores_df[col_name] = scaler.fit_transform(
scores_df[col_name].values.reshape(-1, 1)).reshape(1, -1)[0]
mod_scores_df[col_name] = \
mod_scores_df[col_name] * accuracy_dict[col_name]
mod_scores_df['Score'] = mod_scores_df.mean(axis=1)
See a Python example that creates doped records, evaluates several detectors, and builds an accuracy-weighted ensemble. section 14.6.4
