Data drift is a change in incoming data or its relationship with the target variable compared with what was observed during training. Monitoring for drift helps identify conditions that may degrade model accuracy and guides decisions about feature engineering, training data, or retraining. section 11.0 section 6.3
For API-based monitoring, compare a reference dataset with a microbatch rather than testing each request individually. A single request is not a suitable comparison sample. A monitoring service can accumulate rows, run a report when the window reaches its configured size, and retain recent data by dropping older rows. section 6.2.3
Listing 6.10 MonitoringService to run drift reports periodically
class MonitoringService:
def __init__(
self,
report: Report,
reference: pandas.DataFrame,
workspace: Workspace,
project_id: str,
window_size: int,
):
self.window_size = int(window_size)
self.report = report
self.new_rows = 0
self.reference = reference
self.workspace = workspace
self.project_id = project_id
self.current = pandas.DataFrame()
def iterate(self, new_rows: pandas.DataFrame):
rows_count = new_rows.shape[0]
self.current = pandas.concat(
[self.current, new_rows],
ignore_index=True
) #1
self.new_rows += rows_count
current_size = self.current.shape[0]
if current_size > self.window_size: #2
self.current = self.current.iloc[-self.window_size :]
self.current.reset_index(drop=True, inplace=True)
if current_size < self.window_size: #3
logger.info(
f"Not enough data for measurement: {current_size} "
f"of {self.window_size}. Waiting more data"
)
return
self.report.timestamp = datetime.datetime.now()
logger.info("Running report")
self.report.run(
reference_data=self.reference,
current_data=self.current,
) #4
self.workspace.add_report(
project_id=self.project_id,
report=self.report
) #5
Shows a MonitoringService pattern for collecting incoming rows, managing a window, and triggering drift reports. section 6.2.3
Drift monitoring complements operational service monitoring: uptime, throughput, latency, and error budgets describe service reliability, while drift checks track changes in model inputs and their relation to outcomes. Tools mentioned for this work include deepchecks and Evidently. section 11.0 section 11.4
