Chapter 5: Orchestrating ML pipelines

Introduction

This introduction explains why MLflow and Feast need pipeline orchestration and previews an income-classification batch inference workflow built with Kubeflow Pipelines.

• Manual workflow coordination gap: Explain that MLflow tracks experiments and Feast manages features, but neither removes the need to manually coordinate training, feature updates, and inference. Describe how Kubeflow Pipelines automates those workflows to make ML systems more scalable and reproducible.
  Why it matters: It explains why orchestration is the next step beyond the tools introduced in chapter 4.
• Income classification pipeline roadmap: Describe the chapter’s practical income-classification example as a batch inference workflow that turns manual steps into reusable pipeline components, from loading data through running model inference.
  Why it matters: It gives you a concrete picture of the workflow the chapter will build.

Key terms: MLflow, Feast, pipeline orchestration, Kubeflow Pipelines (KFP), batch pipeline, model inference, income classification, pipeline components

5.1 Kubeflow Pipelines: Task orchestrator

This section introduces Kubeflow Pipelines as a way to assemble isolated ML tasks into repeatable workflows, using a typical inference pipeline as the example.

• Inference pipeline flow: Describe a typical inference pipeline: retrieve data, preprocess it, load a model, perform inference, and store the results. Explain that KFP can run such a pipeline periodically with runtime parameters.
  Why it matters: The example shows what the pipeline needs to coordinate.
• Component assembly and isolation: Explain that the Python kfp SDK defines pipeline steps called components, which are combined into a Kubeflow pipeline. State that each component runs in its own Kubernetes pod, giving the step a clean, isolated environment.
  Why it matters: This is the basic construction model for the pipelines built later.
• Standardized workflow steps: Explain that packaging ML tasks as Kubeflow components provides a standardized approach intended to help make workflows repeatable, reuse steps across models, and support reliable data flow between steps.
  Why it matters: Standardized components address the workflow problems that motivate using KFP.

Key terms: inference pipeline, runtime parameters, KFP, SDK, components, Kubernetes pod, Kubeflow pipeline, Kubeflow components, repeatable, standardize steps

5.1.1 Kubeflow components

This section builds four reusable Kubeflow components for an income-classifier inference pipeline and explains how to package, connect, and design them.

• Four component inference flow: Trace the pipeline from reading user IDs in MinIO, through retrieving features from Feast and a model from MLflow, to writing predictions back to MinIO. Explain that each task is a separate Kubeflow component.
  Why it matters: The sequence shows what data each component needs to pass to the next.
• Component packaging and specification: Describe how a task becomes a Kubeflow component: write its Python logic, build and push a Docker image containing its code and dependencies, then define its image, command, arguments, and typed inputs and outputs in component.yaml. Explain that the component executes in a Kubernetes pod and that dataset output paths let subsequent components use its results.
  Why it matters: A component needs both executable code and an interface that Kubeflow can connect to other steps.
• Cluster ready Feast features: Explain why the locally registered Feast feature sources must be registered again with a MinIO endpoint reachable inside the Kubernetes cluster. Describe how the feature component loads its feature-store configuration and calls get_historical_features with an entity DataFrame and feature list.
  Why it matters: A feature lookup inside a pipeline cannot rely on the localhost address used during local setup.
• Registered model prediction step: Describe how the inference component uses MLflow to obtain the registered model’s URI, loads it with a method appropriate to its framework, runs predict_proba on the retrieved features, and writes the predictions to its dataset output for the separate write-data component.
  Why it matters: Keeping prediction and storage separate makes the inference step easier to reuse.
• Reusable component design principles: Justify designing components with one responsibility, explicit inputs and outputs, configurable parameters, and documentation. Relate those choices to reuse across pipelines, collaboration on separate steps, and easier isolated testing and debugging.
  Why it matters: These choices make components useful beyond a single working pipeline.

Key terms: MinIO, Feast, MLflow, Kubeflow component, component.yaml, Docker image, Dataset, inputPath, outputPath, data_output_path

5.1.2 Income classifier pipeline

This section builds and runs an income classifier workflow in Kubeflow Pipelines, from connecting component tasks to inspecting a completed run.

• Pipeline tasks and dependencies: Explain how the income classifier pipeline represents its four tasks as a directed acyclic graph, with task outputs passed to downstream inputs to establish execution dependencies.
  Why it matters: The connections determine both where data goes and when each task can run.
• Pipeline definition and compilation: Describe how the Python pipeline loads component definitions from files, uses @dsl.pipeline to define a function with runtime-adjustable parameters, creates tasks by supplying component inputs, and compiles that function to YAML for Kubeflow.
  Why it matters: This turns reusable components into a workflow that Kubeflow can accept and execute.
• Pipeline upload and execution: Trace the UI workflow from uploading the compiled YAML and viewing its visualization to creating an experiment, starting a run with runtime parameters, and checking its progress and resulting predictions.
  Why it matters: Defining a pipeline is only useful once it can be submitted, tracked, and checked for output.
• Batch inference tool boundary: Distinguish KFP’s use here for orchestrating batch inference workflows from the real-time inference and production model deployment that the next chapter addresses with BentoML.
  Why it matters: It clarifies why completing this pipeline does not yet provide a real-time prediction service.

Key terms: directed acyclic graph (DAG), task.outputs, fetch_data_task, retrieve_features_task, run_inference_task, write_data_task, load_component_from_file, @dsl.pipeline, pipeline parameters, compiler

5.2 Summary

The section summarizes how Kubeflow Pipelines orchestrates ML workflows by connecting reusable components and managing their execution.