A containerized Python component runs its code and dependencies from a built container image within a Kubeflow pipeline. It occupies the middle ground between lightweight Python components and full container components. section 7.3.2
Use this pattern when operational repeatability matters:
- Set target_image to build and push an image, then run the component without installing dependencies at startup. section 7.3.2
- Keep packages_to_install in the component declaration so dependencies are resolved during image construction. section 7.3.2
- Import methods, symbols, and code from other Python modules instead of defining everything inside the component function. section 7.3.2
- Pass Input and Output artifacts between components using their path attributes. section 7.3.2
MovieLens data-preparation pipeline
A Python 3.11 component downloads MovieLens data with requests, while related components unzip files, convert CSV to Parquet, split datasets, upload artifacts to MinIO, and run data QA. The components are then connected into a Kubeflow pipeline.
Listing 7.24 Dataset download function with single output
from kfp.dsl import Output
@dsl.component(
base_image="python:3.11",
target_image="mlsysfromscratch/data_preprocessor:1.0.0",
packages_to_install=["requests"])
def download_ml25m_data(output_path_one: Output[Artifact]):
import requests
url = 'https://files.grouplens.org/datasets/movielens/ml-25m.zip'
response = requests.get(url, stream=True, verify=False)
with open(output_path_one.path, 'wb') as file:
for chunk in response.iter_content(chunk_size=1024*1024):
if chunk:
file.write(chunk)
Inspect the component declaration and runnable download implementation. section 7.3.2
