Data passing transfers parameters and data artifacts between executable stages of an ML pipeline. In Kubeflow Pipelines (KFP), each component runs in an isolated Kubernetes pod, so components need explicit mechanisms to exchange results and inputs. section 7.2
A production-oriented KFP data-passing flow separates three responsibilities:
- Declare component interfaces with typed inputs and outputs. section 7.2.1
- Define the runtime environment, including the base image and any packages installed for the component. section 7.2.1
- Connect component outputs to downstream inputs in the pipeline DAG; KFP handles serialization and data passing for simple values. section 7.2.1
- Compile the pipeline to YAML, where generated boilerplate handles argument parsing, serialization, and execution. section 7.4
Listing 7.5 Kubeflow components and pipeline definition
import kfp.dsl as dsl
@dsl.component( #1
packages_to_install=["pyjokes"], #2
base_image="python:3.11" #3
)
def generate_joke() -> str:
import pyjokes
return pyjokes.get_joke()
@dsl.component
def count_words(input: str) -> int:
return len(input.split())
@dsl.component
def output_result(count: int) -> str:
return f"Word count: {count}"
@dsl.pipeline( #4
name="joke_pipeline", #4
description="simple pipeline to demo data passing") #4
def pipeline():
generate_joke_op = generate_joke()
count_word = count_words(input=
generate_joke_op.output) #5
output_result = output_result(count=
count_word_op.output) #5
if __name__ == '__main__':
from kfp import compiler
compiler.Compiler().compile(
pipeline_func=pipeline,
package_path='_pipeline.yaml'
)
Inspect the compact KFP definition that creates a joke, passes its output to a word counter, and passes the count to a formatter. section 7.2.1
Simple-value pipeline
A `generate_joke` component returns a string. `count_words` accepts that string and returns an integer. `output_result` accepts the integer and returns formatted text. The component functions remain small, while the pipeline definition expresses the data dependencies between stages. section 7.2.1
