A time-based cache prevents repeated data ingestion when a regularly updated dataset is still fresh enough for the next machine-learning workflow run. section 5.4.2 section 7.5.2
Implement the freshness check by storing:
- the location of the ingested and cleaned dataset, such as a remote database location; section 5.4.2
- the dataset's last-updated timestamp; and section 7.5.2
- a freshness window that determines whether the workflow skips ingestion or re-ingests the data. section 5.4.2
Two-week freshness window
When a workflow is triggered, it reads the cached timestamp. If the dataset was updated within the past two weeks, training uses the stored dataset directly; otherwise, the workflow reconstructs and executes data ingestion. section 5.4.2 section 7.5.2
Figure 5.22 The workflow has been triggered, and we check whether the data has been updated within the last two weeks by accessing the cache. If the data is fresh, we skip the execution of the unnecessary data ingestion step and execute the model training step directly.
The workflow diagram traces the cache lookup, freshness decision, skipped ingestion step, and direct transition to model training. section 5.4.2
