Argo Workflows is an open-source, container-native engine for orchestrating parallel workflows and tasks on Kubernetes. It fits systems that need repeatable job pipelines, such as checking for new data, transforming it, saving results to a database, and notifying a team. section 9.3 section 9.3.2
A typical Argo deployment organizes execution around these responsibilities:
- Define a workflow and its steps as a Kubernetes custom resource definition, usually represented by YAML. section 9.3.2
- Submit the workflow to Argo, whose controller creates the required resources in the Kubernetes cluster. section 9.3.2
- Run each task in a separate Kubernetes pod, isolating execution by container and pod. section 9.3.2
- Pass inputs and outputs through files and dependent artifacts mounted into the consuming step’s container. section 9.3.2
- Debug runs with either Argo commands or standard Kubernetes CLI commands. section 9.3.2
Data-processing workflow
A workflow can check for new data, pass discovered paths to a transformation step, save transformed data into a database, and notify the data-science team through Slack. The YAML uses a Workflow object, an entrypoint, templates, sequential steps, and artifact arguments. section 9.3.2
Listing 9.2 A sample workflow for Argo Workflows with a series of steps
apiVersion: argoproj.io/v1alpha1
kind: Workflow #1
metadata:
generateName: data-processing-
spec:
entrypoint: argo-steps-workflow-example
templates:
- name: argo-steps-workflow-example
Steps: #2
- - name: check-new-data
template: data-checker #3
- - name: transform-data
template: data-converter
arguments:
artifacts:
- name: data-paths #4
from: "{{steps.check-new-data.outputs.
artifacts.new-data-paths}}" #5
- - name: save-into-db
template: postgres-operator
- - name: notify-data-science-team
template: slack-messenger
- name: data-checker #6
container:
image: docker/data-checker:latest
command: [scan, /datastore/ds/]
outputs:
artifacts:
- name: new-data-paths #7
path: /tmp/data_paths.txt
- name: data-converter
inputs:
artifacts:
- name: data_paths #8
path: /tmp/raw_data/data_paths.txt
container:
image: docker/data-checker:latest
command: [data_converter, /tmp/raw_data/data_paths.txt]
- name: save-into-db
.. .. ..
- name: notify-data-science-team
.. .. ..
Inspect a representative YAML workflow showing the Workflow resource, entrypoint, templates, ordered steps, and artifact transfer. section 9.3.2
