A container image packages training code and its dependent libraries so a training service can run that code as a container rather than directly as a process. The service treats the image as a black box: it can execute different training implementations through one standard launch mechanism while avoiding per-job environment and package setup. section 3.2.2
This boundary decouples algorithm development from execution operations. A training request identifies an image and tag; the service queues the request, pulls the image, launches a container, and tracks the resulting job. A communication protocol still defines inputs and outputs such as datasets, hyperparameters, model locations, metrics locations, and data formats. section 3.2.2 section 3.3.2
A production image workflow commonly includes:
- Write training code and build it into a Docker image. section 3.3.2
- Publish the image to an artifact repository so the training service can pull it. section 3.3.2
- Automate builds and publication after code is checked into Git, for example with a Jenkins pipeline and JFrog Artifactory. section 3.3.2
- Run the image as a container; execute its training code through a bash script and upload metrics and the final model from the container. section 3.3.2
Image-based training job
A data scientist submits a request naming a Docker image and tag. The service creates a queued job, returns an ID for progress tracking, and starts a container in the local Docker engine. The running code reports training metrics to a metadata store and saves the final model when training completes. section 3.3.2
Container images also make workflow steps portable across workers. In Argo Workflows, a template can name the image for a step, while artifacts pass files between steps. Packaging each step as an image provides isolation and lets workers execute code without separately configuring their environments. section 9.3.2
Listing 9.2 A sample workflow for Argo Workflows with a series of steps
apiVersion: argoproj.io/v1alpha1
kind: Workflow #1
metadata:
generateName: data-processing-
spec:
entrypoint: argo-steps-workflow-example
templates:
- name: argo-steps-workflow-example
Steps: #2
- - name: check-new-data
template: data-checker #3
- - name: transform-data
template: data-converter
arguments:
artifacts:
- name: data-paths #4
from: "{{steps.check-new-data.outputs.
artifacts.new-data-paths}}" #5
- - name: save-into-db
template: postgres-operator
- - name: notify-data-science-team
template: slack-messenger
- name: data-checker #6
container:
image: docker/data-checker:latest
command: [scan, /datastore/ds/]
outputs:
artifacts:
- name: new-data-paths #7
path: /tmp/data_paths.txt
- name: data-converter
inputs:
artifacts:
- name: data_paths #8
path: /tmp/raw_data/data_paths.txt
container:
image: docker/data-checker:latest
command: [data_converter, /tmp/raw_data/data_paths.txt]
- name: save-into-db
.. .. ..
- name: notify-data-science-team
.. .. ..
Inspect a sequential Argo workflow in which templates select containerized step implementations and artifacts carry outputs into later steps. section 9.3.2
