Chapter 7: Data analysis and preparation

Introduction

This introduction explains why data preparation matters and previews how Kubeflow notebooks and pipelines will be used for two capstone ML projects.

• Data preparation as foundation: Explain why the chapter emphasizes data preparation even as model architectures change: the quality and integrity of input data shape the reliability of model results.
  Why it matters: A strong model cannot compensate for unreliable input data.
• Kubeflow preparation workflow: Describe the chapter’s path from simpler examples of interactive analysis in Kubeflow notebooks and data passing between pipeline components to reproducible data preparation pipelines for the projects.
  Why it matters: The simpler examples establish the workflow used in the capstone projects.
• Two capstone data projects: Distinguish the chapter’s two applications: an ID card detector using a YOLO variant for object detection, and a MovieLens movie recommender whose preparation introduces tabular data handling and combines six components into a pipeline.
  Why it matters: The two projects show how data preparation applies to different kinds of ML data.

Key terms: data preparation, garbage in, garbage out, data quality, data integrity, Kubeflow notebooks, toy jokes pipeline, data passing, Kubeflow components, data preparation pipeline, reproducibility

7.1 Data analysis

This section explains the infrastructure needs of interactive ML data analysis and introduces Kubeflow notebooks as a way to meet them alongside production pipelines.

• Data analysis infrastructure needs: Explain why data analysis needs access close to large datasets, on-demand interactive compute such as GPUs, and an experiment environment that resembles deployment while supporting versioning and CI/CD.
  Why it matters: These constraints explain why a notebook alone is not enough for production-oriented ML work.
• Kubeflow notebook approach: Describe how Kubeflow notebooks provide familiar browser-based tools on the same cluster as pipelines, and how shared base images and the same CI build process help align notebook, pipeline, and deployment environments.
  Why it matters: It connects interactive exploration to the platform used to build and deploy ML systems.

Key terms: data analysis, ad hoc interactive access, GPUs, deployment environment, CI/CD, Kubeflow notebooks, Jupyter Notebooks, VS Code, RStudio, base image

7.1.1 Launching a notebook server in Kubeflow

This section shows how to create a Kubeflow notebook and choose its namespace, image, and compute resources without unnecessarily tying up cluster capacity.

• Kubeflow notebook creation: Describe how to start creating a notebook from the Kubeflow UI and choose the image that will run on the cluster.
  Why it matters: These are the first choices needed to launch a notebook for data analysis.
• Notebook namespace implications: Explain why the Kubernetes Namespace selector matters when creating a notebook, including the kinds of access or limits it can affect.
  Why it matters: The chosen namespace can determine which resources and data the notebook can use.
• Notebook compute resource choices: Explain how notebook resource settings relate to Kubernetes requests and limits, when to select GPUs or other accelerators, and why idle notebooks should release expensive accelerator resources.
  Why it matters: Careful resource choices help notebooks run reliably without wasting scarce cluster capacity.

Key terms: Notebooks, NEW NOTEBOOK, Image, Kubernetes (K8s) Namespace, resource quotas, volume mounts, secrets, limits, requests, GPUs

7.1.2 Workspace and data volumes

Explains how Kubeflow notebooks use persistent workspace storage and optional data volumes, including when existing storage can be reused.

• Notebook workspace and data volumes: Distinguish the workspace volume Kubeflow mounts automatically at /home/jovyan from optional data volumes, and explain how each is typically provided. Describe how selecting an existing workspace volume preserves workspace data when changing the running container.
  Why it matters: The distinction helps you keep notebook settings and data available across notebook changes.
• Persistent Volume Claim meaning: Explain that a Persistent Volume Claim is a containerized application’s request for storage on a Kubernetes cluster, allowing it to access storage volumes provisioned and managed by the underlying infrastructure.
  Why it matters: Understanding a PVC clarifies what it means to mount an existing data volume in a notebook.

Key terms: workspace volume, data volumes, default storage class, Persistent Volume Claims (PVCs), /home/jovyan, Persistent Volume Claim (PVC), Persistent Volumes (PVs)

7.1.3 Configurations and affinity/tolerations

Explains how Kubeflow notebook launch settings configure a pod, influence its placement, and affect what happens when the notebook resource is changed.

• PodDefault notebook configurations: Explain that the notebook UI’s Configurations are PodDefaults in the namespace and that selecting one applies its settings to the notebook pod at launch. Describe the example that injects MLFLOW_TRACKING_URI and how PodDefaults can also provide credentials, setup, or controlled volume mounts.
  Why it matters: PodDefaults let users select cluster-managed settings without configuring each notebook pod by hand.
• Affinity and toleration scheduling: Distinguish how affinity attracts a notebook pod to matching nodes while tolerations permit it onto nodes whose taints would otherwise keep it away. Explain their use in controlling access to nodes with special resources such as GPUs.
  Why it matters: These controls help place notebooks on appropriate cluster hardware.
• Notebook launch and resource changes: Describe what launching the notebook sets in motion: optional shared-memory settings, workspace provisioning, pod scheduling, image pulling, and Kubeflow-managed networking. State that changing the launched notebook’s Kubernetes resource triggers redeployment and removes nonpersistent data when the container restarts.
  Why it matters: Recognizing what launch and redeployment do helps avoid losing work stored only in a notebook container.

Key terms: Configurations, PodDefaults, add-proxy-vars, env, MLFLOW_TRACKING_URI, affinity, tolerations, taints, GPUs, shared memory

7.1.4 Customizing the menu

Cluster administrators customize the Kubeflow notebook menu through a ConfigMap, controlling available images and notebook scheduling choices.

• Notebook menu ConfigMap: Explain that the notebook menu’s defaults and available choices come from a ConfigMap in the Kubeflow namespace, usually named with a jupyter-web-app-config prefix. Describe how default values and readOnly settings affect what users can change.
  Why it matters: This is where an administrator controls the choices users see when launching notebooks.
• Notebook image choices: Describe how image groups define the default image and drop-down options for Jupyter, code-server, and RStudio notebooks. Explain that administrators can update the listed images through a CD workflow and can restrict users to listed images.
  Why it matters: Controlled image choices help make notebook environments reproducible and similar to deployment environments.
• Notebook node scheduling choices: Explain how affinity options can offer users CPU-node choices such as small-cpu and xl-cpu while defaulting notebooks to smaller-CPU nodes. Distinguish this scheduling preference from restricting access, which requires node taints together with toleration configurations.
  Why it matters: The distinction lets administrators conserve powerful nodes without mistaking a scheduling preference for an access restriction.

Key terms: ConfigMap, jupyter-web-app-config, spawnerFormDefaults, readOnly, image, imageGroupOne, imageGroupTwo, value, options, CD pipeline

7.1.5 Creating a custom Kubeflow notebook image

This section explains what a custom Kubeflow notebook image contains, how the provided Dockerfile builds one, and why custom images help teams move from local notebooks to Kubeflow.

• Custom notebook image layers: Describe the dependencies, code-server or Jupyter runtime, and configuration files that make up a custom Kubeflow notebook image, including the configuration needed to launch the server compatibly with Kubeflow.
  Why it matters: These layers explain what must be present for a custom notebook image to run in Kubeflow.
• Multistage image build: Explain that the project repository provides a multistage Dockerfile that accepts a base image as an argument and can build a code-server or Jupyter image. State that an organization should review and adjust it to optimize image size.
  Why it matters: The provided Dockerfile is a starting point, not necessarily an image optimized for an organization's deployment.
• Notebook workflow transition: Explain why familiar notebook interfaces and custom images make it easier for teams to move from local development to Kubeflow, where they can launch notebooks and run experiments in a modern Git-based workflow.
  Why it matters: The point of customization is to make the move to a platform-based ML workflow easier for its users.

Key terms: custom image, code-server, Jupyter, configuration files, Kubeflow, multistage Dockerfile, base image, image size, Kubeflow notebooks, local development workflow

7.2 Data passing

Introduces why data passing matters in Kubeflow Pipelines: components run in separate Kubernetes pods, so they need a way to share data in an ML workflow.

• Data passing between isolated components: Explain that each KFP component runs in a separate Kubernetes pod, so components in an ML workflow need a mechanism to pass and share data.
  Why it matters: Pod isolation explains why connecting pipeline components requires deliberate data passing.

Key terms: KFP, components, K8s pod, data passing

7.2.1 Scenario 1: Passing simple values to downstream components

This section shows how a Kubeflow pipeline passes simple return values between components and explains how those components are packaged and executed.

• Simple value pipeline flow: Trace how generate_joke produces text, count_words receives it and returns a word count, and a final component prints that count. Explain that a returning component exposes its value through its output attribute for a downstream component.
  Why it matters: This is the basic pattern for passing small values between Kubeflow components.
• Component dependency packaging: Describe using @dsl.component to turn functions into components, keeping imports inside their functions, and making non-core Python packages available in each component's container through its image or packages_to_install. Distinguish those explicit options from installing a package inside the function, which the section does not recommend.
  Why it matters: Each component runs in its own container, so its code must have the dependencies it needs there.
• Compiled component execution structure: Explain how the compiled pipeline YAML separates a component's input/output interface, its deployment specification for the container and commands, and its DAG task for workflow connections. Describe how the generated commands prepare the environment, write component code to a temporary file, and run it through the KFP executor, with injected helper code handling returned values and data passing.
  Why it matters: The generated YAML makes visible how KFP runs isolated functions and passes their results onward.

Key terms: generate_joke, count_words, output attribute, KFP, @dsl.component, packages_to_install, base_image, pyjokes, Docker container, joke_pipeline.yaml

7.2.2 Scenario 2: Passing paths for larger data

This section explains why KFP components should pass paths to file-backed artifacts for larger data, shows that pattern in a three-component pipeline, and contrasts KFP v2 artifact types with v1 path conventions.

• Pod memory and data passing: Explain why passing large raw values between KFP components is risky and why storing the data in a file or directory and passing its path is recommended.
  Why it matters: Each component runs in a Kubernetes pod with a memory limit, so large in-memory data can cause an OOMKilled error.
• Artifact-backed component handoff: Trace how the joke example writes data through an Output[Artifact], reads it through an Input[Artifact], and keeps a small final string as a direct output.
  Why it matters: The example shows where file handling belongs in a KFP v2 component chain and where it is unnecessary.
• KFP version migration conventions: Distinguish KFP v1's file and path parameter naming conventions from KFP v2's typed Input and Output artifacts, and describe the migration changes stated in the aside.
  Why it matters: The distinction helps prevent v1 naming rules from being applied to v2 components.

Key terms: K8s pod, memory limit, OOMKilled, file or directory, path, Output[Artifact], Input[Artifact], generate_joke, count_words, output_result

7.2.3 Overview of KFP v2 artifact types

This section explains how to choose KFP v2 artifact types for pipeline data, models, metrics, reports, and other outputs.

• Typed pipeline artifact system: Explain that KFP v2 artifact types give pipeline data semantic meaning, standardized storage handling, and metadata tracking, supporting data lineage and model provenance.
  Why it matters: Choosing a meaningful artifact type makes pipeline outputs easier to track and maintain.
• Dataset artifact use: Describe when to use Dataset for raw or processed ML data, feature sets, or data splits, and how a component can consume Input[Dataset] and produce Output[Dataset] with metadata.
  Why it matters: Dataset artifacts give components a consistent way to exchange and describe ML data.
• Model artifact provenance: Describe using Model for trained models, checkpoints, weights, or pretrained model artifacts, including a component’s Output[Model] and metadata that helps trace training data and parameters.
  Why it matters: A model artifact helps connect a trained output to how it was produced.
• Numerical and classification metrics: Distinguish Metrics for simple numerical measurements from ClassificationMetrics for classification-specific outputs such as confusion matrices and ROC curves; describe SlicedClassificationMetrics as applying classification metrics to a data slice.
  Why it matters: The appropriate metrics type makes evaluation results easier to compare and inspect.
• Rendered report artifact choices: Distinguish HTML for rich or interactive visualizations and reports from Markdown for readable documentation, simple reports, model cards, and summaries; note that both can render in the KFP UI.
  Why it matters: The format determines how people can read and inspect a pipeline’s reports.
• Generic Artifact fallback: Explain when to use the base Artifact type for outputs that do not fit a specialized type, such as generic files, intermediate results, or configuration files.
  Why it matters: The base type lets a pipeline handle unusual outputs without forcing them into a misleading category.

Key terms: KFP v2, artifact types, metadata, data lineage, model provenance, Dataset, Input[Dataset], Output[Dataset], data splits, Model

7.3 Data preparation in action

This section previews how data preparation for two capstone projects will become a Kubeflow pipeline built from reusable components.

• Data preparation pipeline structure: Describe the planned data preparation workflow for both capstone projects: download each dataset, split it into training, test, and validation sets, and assemble the two Kubeflow components into one executable pipeline.
  Why it matters: This workflow provides the data that the capstone training pipelines will use.
• Component design learning focus: Explain that the chapter aims not only to create a first Kubeflow Pipeline (KFP), but also to practice making components composable and well-defined and customizing them for particular needs.
  Why it matters: Reusable, clearly defined components are a broader goal than getting one pipeline to run.

Key terms: data preparation, Kubeflow component, training, test, and validation datasets, executable pipeline, KFP, composable, well-defined Kubeflow components, customize

7.3.1 Data preparation: Object detection

This section builds and runs a Kubeflow data-preparation pipeline that downloads an ID-card dataset, organizes it for YOLOv8, splits it, and checks the results.

• YOLOv8 dataset layout: Describe the configuration and file layout YOLOv8 expects: dataset root, training/validation/test image locations, matching labels, and indexed class names.
  Why it matters: The splitting component must produce data that the later YOLOv8 training step can read.
• Dataset acquisition and storage: Trace how the download component fetches the filtered MIDV-500 ID-card images and labels, extracts the tar archive, and uploads the resulting files to a MinIO bucket.
  Why it matters: A shared object store makes the prepared source data available to pipeline components.
• Component packaging and artifact passing: Explain how Python functions become self-contained KFP components and how Dataset output and input paths let downstream components access data without passing a large dataset as an ordinary value.
  Why it matters: Component dependencies and artifact paths let Kubeflow connect stages without manually coordinating their execution.
• Reproducible dataset splitting: Describe how the splitting component pairs images with labels, uses two train_test_split operations to make reproducible 75/15/10 training, validation, and test partitions, and organizes each partition into image and label directories.
  Why it matters: A repeatable split preserves image–label matches and produces the structure required for training and validation.
• Pipeline execution and inspection: Trace how the download, split, and verification components are connected, compiled to YAML, uploaded and run in Kubeflow, then checked through the graph and component logs; explain why production runs need log cleanup or rotation.
  Why it matters: Running and inspecting the pipeline confirms that its dependencies and prepared files work as intended without letting recurring logs exhaust storage.

Key terms: Ultralytics YOLO format, images, labels, names, id_card, MIDV-500, tarfile, MinIO, bucket, boto3

7.3.2 Data preparation: Movie recommender

This section builds a MovieLens data preparation pipeline in Kubeflow, using reusable components to produce Parquet datasets, upload them to MinIO, and check the results.

• Movie recommender pipeline stages: Trace the six components from downloading and unzipping MovieLens data through CSV-to-Parquet conversion, splitting, MinIO upload, and quality assurance. Explain that the components are kept separate to demonstrate Kubeflow data passing, although a production pipeline might combine some of them.
  Why it matters: The stage sequence shows where the training data comes from and where errors can be caught.
• Kubeflow artifact path handoffs: Explain how components pass file and folder artifacts by writing to output paths and reading subsequent input paths. Describe how Kubeflow assigns those paths, persists artifacts through MinIO between components, and why the upload component supplies a desired filename separately.
  Why it matters: Path ownership and artifact persistence determine whether downstream components receive the intended files.
• Parquet conversion and reuse: Justify converting the two tabular CSV files to Parquet, and describe how the same conversion component is reused for movies and ratings. State that this example demonstrates conversion with pandas and fastparquet rather than sharding or optimizing the resulting files.
  Why it matters: The format choice supports larger tabular datasets without implying optimizations the example does not perform.
• Containerized Python component behavior: Distinguish the download component’s containerized Python approach from lightweight Python components: setting target_image builds and pushes an image with dependencies installed at build time, so runtime startup avoids installing them. Mention that this approach also permits imports from other modules and requires a working Docker build-and-push environment.
  Why it matters: Build-time packaging helps when runtime dependency installation is slow or unreliable.
• Prepared data quality checks: Describe the QA component’s checks that four expected files exist and that the training dataset is about 75% of the full ratings dataset. Explain why checking generated data early matters, while recognizing that production checks should also validate assumptions about size, schema, and content.
  Why it matters: Early validation catches missing or incorrectly sized outputs before training has to diagnose them.
• Notebook pipeline submission: Describe compiling and uploading the pipeline with the Kubeflow SDK from a notebook running on the cluster, where injected environment settings spare the client from an explicit endpoint or credentials. Distinguish the uploaded pipeline’s UUID from the experiment ID and run name needed for a run, and explain why the example starts its run in the UI under a multiuser setup.
  Why it matters: The submission workflow separates defining a pipeline from identifying it and starting an authorized run.

Key terms: MovieLens 25M, Dataset download component, Data unzip component, CSV to Parquet conversion component, Dataset split component, MinIO upload component, Dataset artifacts, path attribute, InputPath, OutputPath

7.4 Summary

The chapter summary connects Kubeflow notebooks, component data passing, and pipeline assembly to the data preparation needed for the object detector and movie recommender projects.