Evidently is an open-source tool for monitoring machine-learning data and detecting drift between historical reference data and current inference data. It supports production reliability by revealing distribution changes before they degrade model performance. section 2.1.6 section 6.2
A drift-monitoring workflow with Evidently typically includes:
- Compare a reference dataset representing normal training behavior with newly collected inference data. section 6.2
- Run suitable statistical tests, such as Kolmogorov–Smirnov, chi-square, or Wasserstein distance. section 6.2
- Publish metrics and reports to an Evidently workspace or dashboard for operational inspection. section 6.2.2
- Use the result as a pipeline control: stop inference when drift is detected or trigger retraining workflows. section 2.1.6 section 6.2.2
Income-classifier inference pipeline
A Kubeflow inference pipeline retrieves features from Feast, compares them with the training reference dataset, generates an Evidently report, and stops execution if significant drift appears. Each run updates the dashboard, where drift and data-quality metrics can be inspected over time. section 4.0 section 6.2.2
Listing 6.9 KFP component for detecting data drift
def detect_drift(
model_name: str,
model_stage: str,
mlflow_host: str,
minio_host: str,
access_key: str,
secret_key: str,
reference_dataset_name: str,
feature_dataset_path: str,
evidently_workspace_url: str,
evidently_ui_project_name: str,
):
os.environ["AWS_ACCESS_KEY_ID"] = access_key
os.environ["AWS_SECRET_ACCESS_KEY"] = secret_key
if not minio_host.startswith("http"):
os.environ["AWS_ENDPOINT_URL"] = "http://" + minio_host
else:
os.environ["AWS_ENDPOINT_URL"] = minio_host
mlflow.set_tracking_uri(mlflow_host)
mlflow_client = MlflowClient(mlflow_host) #1
evidently_workspace = \
RemoteWorkspace(evidently_workspace_url) #2
model_run_id = None
for model in mlflow_client.search_model_versions(f"name='{model_name}'"):
if model.current_stage == model_stage:
model_run_id = model.run_id
break
if not model_run_id:
raise ValueError(
f"Model in stage {model_stage} "
f"not found for {model_name}."
)
run = mlflow.get_run(model_run_id)
dataset_source = None
for dataset_input in run.inputs.dataset_inputs:
for tag in dataset_input.tags:
if tag.value == reference_dataset_name:
dataset_source = json.loads(
dataset_input.dataset.source
)["uri"]
break
if not dataset_source:
raise ValueError(
f"Reference dataset {reference_dataset_name} "
"not found."
)
bucket_name = dataset_source.split("/")[0]
object_name = "/".join(dataset_source.split("/")[1:])
file_path = object_name.split("/")[-1]
download_file_from_minio(
minio_host=minio_host,
access_key=access_key,
secret_key=secret_key,
bucket_name=bucket_name,
object_name=object_name,
file_path=file_path,
) #3
reference_df = pd.read_csv(file_path)
feature_df = pd.read_parquet(feature_dataset_path) #4
feature_df.drop(columns=["user_id"], errors="ignore", inplace=True)
report = Report(
metrics=[
DatasetDriftMetric(),
DatasetMissingValuesMetric(),
ColumnDriftMetric(column_name="Education"),
ColumnSummaryMetric(column_name="Education"),
ColumnDriftMetric(column_name="Marital-Status"),
ColumnSummaryMetric(column_name="Marital-Status"),
ColumnDriftMetric(column_name="Native_country"),
ColumnSummaryMetric(column_name="Native_country"),
ColumnDriftMetric(column_name="Occupation"),
ColumnSummaryMetric(column_name="Occupation"),
ColumnDriftMetric(column_name="Race"),
ColumnSummaryMetric(column_name="Race"),
ColumnDriftMetric(column_name="Relationship"),
ColumnSummaryMetric(column_name="Relationship"),
ColumnDriftMetric(column_name="Sex"),
ColumnSummaryMetric(column_name="Sex"),
ColumnDriftMetric(column_name="Workclass"),
ColumnSummaryMetric(column_name="Workclass"),
],
) #5
report.run(
reference_data=reference_df,
current_data=feature_df
) #6
report_file_path = "drift_report.json"
with open(report_file_path, "w") as f:
f.write(report.json())
print("report written")
project = get_evidently_project(
evidently_workspace,
evidently_ui_project_name
)
print("project retreived")
evidently_workspace.add_report(project.id, report) #7
project.save()
Inspect the Python Kubeflow component that retrieves the reference model data through MLflow and MinIO, initializes a remote Evidently workspace, and generates the drift report. section 6.2.2
