Step memoization skips a workflow step when cached results or metadata show that rerunning it would be redundant. It improves end-to-end performance by avoiding costly work such as re-ingesting unchanged data or retraining an unchanged model. section 5.4.2 section 5.4.3
A cache can be time-based or content-based. A time-based cache treats data as fresh within a chosen window; a content-based cache stores extracted information, such as record counts, model artifacts, hyperparameters, or performance statistics, to decide whether change is significant enough to require execution. section 5.4.2 section 5.4.3
Skipping data ingestion
When a workflow runs again, it can check a cache containing the ingested dataset location and its update timestamp. If the dataset was updated within two weeks, it skips ingestion and sends the existing dataset directly to model training. section 7.5.2 section 5.4.2
Listing 9.40 Memoization for the data ingestion step
- name: data-ingestion-step
serviceAccountName: argo
memoize:
key: "step-cache"
maxAge: "1h"
cache:
configMap:
name: my-config
key: step-cache
container:
image: kubeflow/multi-worker-strategy:v0.1
imagePullPolicy: IfNotPresent
command: ["python", "/data-ingestion.py"]
An Argo Workflows step configures memoization with a cache key, maximum age, and Kubernetes ConfigMap location. A later run within that age can skip the step. section 9.4.2
Cache design and cleanup are operational requirements. High workflow volume can create many caches, so retain useful entries while periodically deleting unused ones, for example by tracking last-hit timestamps. section 5.4.3
In Argo Workflows, a first execution creates the cache; a matching later execution reports a cache hit and completes the memoized step instantly. Cache entries are stored as Kubernetes ConfigMap objects with node, output, creation, and last-hit information. section 9.4.2
