Chapter 9: Model training and validation: Part 2

Introduction

The chapter introduces production-oriented ways to manage pipeline data, observe training, and trace model versions in the ID card detection and movie recommendation projects.

• Production pipeline data management: Explain why the chapter moves beyond chapter 8’s basic training pipelines and introduces Kubernetes PersistentVolumes to manage large datasets in pipelines.
  Why it matters: Reliable access to large datasets helps make training pipelines practical for production use.
• Training visibility and model lineage: Distinguish TensorBoard’s role in making training visible from MLflow’s role in experiment tracking, lineage, and model versioning. Explain why traceability and asynchronous observability matter as models and stakeholders increase.
  Why it matters: These practices make it easier to deliver models to production repeatedly with traceable, deterministic results.

Key terms: Kubernetes PersistentVolumes, PVs, Kubeflow Pipelines Kubernetes SDK, TensorBoard, MLflow, lineage, experiment tracking, model versioning, asynchronous observability

9.1 Storing data with PersistentVolumeClaim

This section introduces shared Kubernetes storage as a way to avoid repeated dataset downloads in Kubeflow pipelines and outlines how the pipeline will manage that storage.

• Shared storage motivation: Explain why the earlier use of stream reads and direct MinIO connections is being replaced with a PersistentVolumeClaim: components can share persistent storage, download a dataset once, and use storage more efficiently.
  Why it matters: The reason for using a PVC makes the pipeline changes that follow easier to understand.
• PVC component lifecycle: Describe the planned use of a library for Kubernetes-specific features in Kubeflow pipelines to create a component that dynamically creates a PVC, mounts it, and finally deletes it.
  Why it matters: The PVC’s creation, mounting, and deletion define how shared storage fits into the pipeline.

Key terms: PersistentVolumeClaim, PVC, MinIO, Kubeflow pipelines, PersistentVolumeClaims component, mount

9.1.1 Refactoring the pipeline with a PVC

The section identifies redundant dataset downloads in the existing pipeline as the problem motivating a refactor.

• Redundant pipeline data downloads: Describe how the Download dataset component fetches the full dataset, the Train model component re-downloads the training, validation, and test splits, and the Validate model component re-downloads the validation split. Explain that this repeated downloading is why the current data handling is not ideal.
  Why it matters: Identifying the repeated downloads makes the reason for refactoring the pipeline clear.

Key terms: Download dataset component, Train model component, Validate model component, training, validation, and test dataset split

9.1.2 Efficient dataset management

This section explains why the pipeline should download a large dataset once and share it through a PVC rather than repeatedly accessing MinIO.

• Single download dataset reuse: Explain that a large dataset should be downloaded once and then accessed through a persistent volume for training and testing, avoiding repeated downloads.
  Why it matters: Reusing one downloaded copy makes the dataset readily available across the pipeline without repeated transfers.
• PVC for shared filesystem access: Explain that MinIO is not a traditional POSIX filesystem, so a PVC provides a more flexible way for pipeline components to work with the dataset. Describe how components depend on one PVC created for the pipeline by a KFP VolumeOp.
  Why it matters: This explains both the reason for introducing the PVC and how it becomes a shared pipeline dependency.

Key terms: dataset, PV, training and testing, MinIO, POSIX, PVC, VolumeOp, createpvc

9.1.3 Creating a VolumeOp

This section shows how to create a pipeline PVC with a VolumeOp before a downstream component uses it.

• Pipeline PVC VolumeOp setup: Describe creating a VolumeOp in the pipeline definition by specifying its name, resource name, access mode, and optionally its volume size. Explain why ReadWriteOnce is a safe mode to choose.
  Why it matters: The pipeline needs the PVC before a downstream component can use it.

Key terms: VolumeOp, Pipeline PVC, ReadWriteOnce, storage classes

9.1.4 Download Op using PVC

This section shows how to mount a previously created PVC in a download component and write the dataset directly to it instead of using MinIO code.

• PVC-backed dataset download: Explain how the download component mounts the previously created PVC and downloads and extracts the dataset into its mount point.
  Why it matters: The mount point is where the downloaded dataset becomes available to later pipeline steps.
• Choice of dataset storage approach: Explain why replacing MinIO code in this component does not make the MinIO approach obsolete.
  Why it matters: The appropriate way to handle data depends on the project's constraints, not just on which example is simpler.

Key terms: kfp.kubernetes.mount_pvc(), PVC, mount point, download_dataset(), MinIO, problem statement, model type, infrastructure limitations

9.1.5 Splitting the dataset directly

The section shows how to split a downloaded dataset on a shared PVC, explicitly ordering the split after the download and moving files into split folders.

• PVC mounting and task order: Explain that split_dataset_op mounts the PVC and must use .after(download_op) so splitting runs after downloading; unlike the earlier Artifact-based pipeline, the shared-volume approach does not give Kubeflow an explicit data dependency from which to infer that order.
  Why it matters: Files on a shared volume must be downloaded before the split task tries to move them.
• Direct splitting on the volume: Describe how the simplified split_dataset_op moves image and label files into the folders for their respective splits within the volume, instead of writing lists of their file names.
  Why it matters: The split folders on the shared volume replace the earlier file-name-list approach.

Key terms: PVC, split_dataset_op, .after(download_op), Artifacts, VolumeOp, images, labels

9.1.6 Simplifying model training

Mounting the dataset PVC simplifies the model-training component by removing dataset downloads and path arguments while retaining YAML creation and model training.

• PVC-based training simplification: Explain how mounting the PVC in train_model_op simplifies train_model: the dataset is available without downloading it from MinIO or passing six InputPath arguments. Describe the remaining path changes: create the data configuration YAML file using the PVC-based path and remove dataset-path references from TRAINING_SCRIPT, while leaving the rest of training unchanged.
  Why it matters: This shows how shared dataset storage reduces training-component code without changing the training task.

Key terms: PVC, mount_pvc, train_model_op, train_model, InputPath, MinIO

9.1.7 Simplifying model validation

This section simplifies model validation by loading model weights from a mounted PVC instead of using MinIO-related code.

• PVC-Based Model Validation: Explain how validation uses the mounted PVC: the validation operation mounts the volume, constructs weights_path from the mount point, and uses that path to initialize the model instead of retrieving weights through MinIO-related code.
  Why it matters: This shows why the PVC makes the training and validation pipeline simpler.

Key terms: PersistentVolumeClaim, PVC, validate_model_op, validate_model, weights_path, mount point

9.2 Tracking training with TensorBoard

The section explains why TensorBoard complements MLflow for training visibility and how the object detection pipeline must expose YOLOv8 logs to Kubeflow’s TensorBoard.

• TensorBoard and MLflow roles: Distinguish MLflow’s high-level experiment and metric tracking from TensorBoard’s detailed tracing of individual training runs and model debugging. Describe the training views TensorBoard offers, such as live metrics, run comparisons, and model visualizations.
  Why it matters: Choosing the right view helps you monitor a long training run without confusing run-level debugging with experiment management.
• YOLOv8 log access in Kubeflow: Explain that YOLOv8 writes TensorBoard-compatible logs to its project directory, even though it is not written in TensorFlow. Describe the proposed pipeline change: pass a VolumeOp, define a mount point, and use that mount point as the project instead of relying on a Kubeflow-created OutputPath.
  Why it matters: TensorBoard’s built-in Kubeflow support still needs access to the logs produced by the training pipeline.

Key terms: MLflow, TensorBoard, training metrics, training runs, YOLOv8, Kubeflow, VolumeOp, mount point, project directory

9.2.1 Launching a new TensorBoard

This section walks through launching a Kubeflow TensorBoard using the PVC from an existing training pipeline run.

• Existing pipeline PVC selection: Explain why a pipeline run must reach the completed create-pvc operation before launching TensorBoard, and how to find the PVC name in the component’s pvc_name row.
  Why it matters: TensorBoard needs the existing volume that holds the training logs.
• TensorBoard creation and connection: Describe how to create a TensorBoard from the Kubeflow sidebar, select the existing PVC, leave Mount Path empty, and connect once the instance is ready.
  Why it matters: These settings attach TensorBoard to the pipeline’s volume so you can inspect its logs.

Key terms: create-pvc, PVC, pvc_name, training logs, TensorBoards, New TensorBoard, PVC Name, Mount Path, CREATE, CONNECT

9.2.2 Exploring YOLOv8’s default graphs

This section shows how to use TensorBoard’s YOLOv8 graphs to assess training performance and inspect the model’s architecture.

• YOLOv8 performance graphs: Explain how TensorBoard’s mean Average Precision (mAP) graphs help assess object detection performance across training epochs and guide changes to training.
  Why it matters: Reading performance trends helps you make evidence-based training decisions rather than guessing.
• Model architecture visualization: Describe how TensorBoard’s architecture graph represents data flow through YOLOv8 and why viewing its layers and connections helps with a complex model.
  Why it matters: The architecture view helps make a complex model’s processing path easier to understand.

Key terms: TensorBoard, YOLOv8, mean Average Precision (mAP), training epochs, model architecture, data flow, layers, connections

9.3 Movie recommender project

This section introduces a movie recommender training workflow that builds on prepared data, rechecks its quality, tracks experiments and model versions, and tests recommendations in a notebook.

• Prepared recommender data: Describe the data available at the start of the project: Parquet files in a cluster MinIO bucket, split into training, test, and validation sets, with rudimentary quality analysis already performed.
  Why it matters: The training workflow starts from prepared data rather than building the data pipeline again.
• Pretraining data revalidation: Explain why the project checks incoming data quality again before training or evaluation: assumptions should be verified at the point of use, allowing data ingestion and transformation to remain separate from training and evaluation.
  Why it matters: A successful check during data creation does not replace checking the data that a training run actually receives.
• Recommender training workflow: Describe the planned workflow: check incoming data, train the recommender, define validation metrics, track experiments and model versions with MLflow, and try the trained model in a notebook. Explain that the emphasis is on reusable training infrastructure rather than recommendation algorithm details, with MLflow presented as an alternative to the Kubeflow tracking used in the object detection example.
  Why it matters: The project shows how an ML pipeline pattern adapts to a different task while supporting continued model improvement.

Key terms: Parquet files, MinIO bucket, train, test, and validation splits, quality assurance, data ingest and transform pipelines, training and evaluation pipelines, model training component, metrics for evaluation, MLflow, model versions

9.3.1 Reading data from MinIO and quality assurance

The section introduces a MinIO data-quality check for the movie recommender and compares reading data directly from MinIO with copying it to a shared volume before training.

• MinIO data quality checks: Describe the initial quality-assurance component for data in MinIO: it checks the expected schema and data types before training, and its checks can be expanded as new defects appear.
  Why it matters: Even simple checks can catch many production data problems before they affect training.
• Training data access trade-offs: Explain why this example reads directly from MinIO rather than first copying the whole dataset to a shared volume, and compare the startup, storage, training-read, and networking trade-offs of both approaches.
  Why it matters: The better data-access approach depends on the dataset and the infrastructure running the training job.
• Old VolumeOp cleanup setting: State that setting TTL_SECONDS_AFTER_WORKFLOW_FINISH in the ml-pipeline-persistenceagent pod automatically clears old VolumeOps after the configured time to live.
  Why it matters: The setting helps prevent old shared volumes from lingering after workflows finish.

Key terms: MinIO, data quality assessment (QA), schema, data types, shared VolumeOp, local volume, network access, TTL_SECONDS_AFTER_WORKFLOW_FINISH, ml-pipeline-persistenceagent, VolumeOps

9.3.2 Model training component

This section introduces the movie recommender’s simple PyTorch model and describes its training and testing loops.

• Recommender model design: Describe the recommender as a PyTorch matrix factorization model with user and item embeddings and two hidden layers. Explain that its simplicity allows quick training, while PyTorch makes the model easier to work with later.
  Why it matters: The model’s design explains what the training component trains and why this implementation was chosen.
• Training and testing cadence: Describe the training loop as using batched inputs and running a testing loop every iteration. Explain that testing less often, such as once every five iterations, can make training faster.
  Why it matters: Testing frequency is a practical way to trade frequent feedback for faster training.

Key terms: matrix factorization, PyTorch, user embeddings, item embeddings, hidden layers, training loop, batched input, testing loop, iteration

9.3.3 Metrics for evaluation

This section defines three example metrics for evaluating a movie recommender and explains why evaluation metrics must reflect business goals.

• Business aligned evaluation metrics: Explain why evaluation metrics should be defined from business goals before modeling and checked to ensure they describe the performance that matters.
  Why it matters: A few metric values can determine whether a model is accepted, so poorly chosen metrics can make a good model look bad or a bad model look good.
• Top fifty recommendation relevance: Distinguish precision from recall for the recommender's top 50 results: precision measures the share of recommended items that are relevant, while recall measures the share of relevant items that were recommended.
  Why it matters: Together, precision and recall describe different ways a list of recommendations can succeed or fail.
• Predicted rating error: Describe RMSE as a measure of error in predicted ratings, with zero representing perfect predictions and lower values indicating better accuracy.
  Why it matters: RMSE assesses rating predictions rather than the relevance or coverage of a recommendation list.

Key terms: evaluation metrics, business goals, stakeholders, toy metrics, precision, recall, true positives, false positives, false negatives, top 50 recommendations

9.3.4 Experiment tracking with MLflow

This section integrates MLflow into movie-recommender training and validation so teams can trace, retrieve, and compare model runs.

• Experiment tracking and tool roles: Explain why comparing runs by memory or a spreadsheet becomes unreliable as experiments scale, and distinguish MLflow’s run indexing and model versioning from TensorBoard’s richer training visualizations.
  Why it matters: Choosing the right tracking tools makes past results reproducible and easier to investigate.
• Model artifact upload paths: Distinguish direct artifact uploads, which require the training environment to access the storage backend, from proxied uploads through the MLflow tracking server; explain the convenience and potential upload bottleneck of proxying.
  Why it matters: The upload path affects both credential setup and performance for large models.
• Training run records and models: Describe how the training code logs hyperparameters, optimizer information, losses, a model summary, and a model with an inferred input/output signature to MLflow, and how loading an earlier model enables a warm restart.
  Why it matters: A central run record connects training choices and results to a reusable model.
• Validation through shared run identity: Trace how a separate validation component uses a run ID to load the trained model directly from MLflow for inference and adds evaluation metrics to the same experiment, even after the training run has completed.
  Why it matters: Shared run identity keeps separately executed training and validation tied to one model history.
• Run comparison and team visibility: Describe comparing a default training run with one using a changed learning rate in MLflow, and explain how run details, tags, descriptions, and the dashboard help a team find and communicate results.
  Why it matters: A shared comparison view makes model performance and its experimental context accessible without manual status updates.

Key terms: experiment tracking, MLflow, TensorBoard, model lineage, model artifact storage, proxied access mode, MLflow tracking server, storage backends, hyperparameters, optimizer

9.3.5 Model registry with MLflow

This section explains how MLflow’s model registry turns logged recommender models into versioned, traceable models that can be promoted through staging toward production.

• Model registry purpose: Explain how a model registry supports version tracking, controlled promotion, lineage, and rollback, while providing a central place and API for managing models.
  Why it matters: These capabilities make it easier to manage and deploy models with confidence in where they came from.
• Registered models and aliases: Trace the path from a model logged with its experiment run to a registered model with a name, versions, and optional aliases and metadata. Describe how a production alias can point to the specific version actively used and be changed for a rollback.
  Why it matters: Keeping versions distinct from the alias that selects one lets teams change the active model without losing its history.
• Staging promotion workflow: Describe the section’s staging workflow: create a staging model placeholder in the MLflow UI, then use a reusable component that accepts a stage name and compares a newly trained model with the current staging model. Distinguish that comparison from promotion to production, which may require manual review or, if an organization chooses, additional empirical tests and automation.
  Why it matters: Separating staging checks from production approval gives teams a place to apply the quality gates they need.

Key terms: model registry, model versions, model lineage, rollback, experiment run, registered model, model version, model alias, staging, production

9.3.6 Creating a pipeline from components

This section assembles the movie recommender components into a compiled Kubeflow pipeline and notes how to split a large pipeline into smaller ones.

• Movie recommender pipeline assembly: Describe how the previously defined functions are compiled into components, connected into a pipeline, and then compiled as a pipeline that uses MinIO for data and MLflow for models.
  Why it matters: This is the step that turns the individual recommender functions into an executable workflow.
• Pipeline composition for complex workflows: Explain why this pipeline has many inputs and how Kubeflow can split such a workflow by letting one pipeline trigger another or be used as a component.
  Why it matters: Pipeline composition offers a way to organize a workflow that does several things without keeping it all in one pipeline.

Key terms: components, pipeline, compile, MLflow, MinIO, inputs, Kubeflow

9.3.7 Local inference in a notebook

The section uses notebook inference to check a registered movie recommender against a user’s viewing history, then connects that check to the broader training pipeline and upcoming production serving work.

• Notebook recommendation inference: Describe how to try the trained recommender in a notebook: reference the model from a run through the model registry and call its recommend method for a specific user. Explain that logged, named input and output tensors with dimensions let a user run inference without the model class or definition.
  Why it matters: This is a controlled way to test the model before serving it in production.
• Recommendation sanity check: Explain why local inference is useful before deployment and how the section checks the example recommendations against the user’s raw viewing history. Describe the user 50 example as a plausibility check, not proof of production readiness.
  Why it matters: Inspecting individual outputs can reveal surprising behavior that aggregate evaluation might not explain.
• Training workflow foundation: Summarize how the chapter’s data pipelines, shared volumes, TensorBoard, and MLflow support a workflow from raw data to trained, tracked, validated models. State that this infrastructure simplifies iteration but does not remove the possibility that further gains will require a model architecture change.
  Why it matters: The notebook check is one step in a repeatable workflow that prepares models for production serving.

Key terms: model registry APIs, input and output tensors, recommend method, local inference, local inference testing, recommendations, viewing history, user 50, ETL, MinIO

9.4 Summary

The chapter summary explains how persistent storage, experiment tracking, and accessible model artifacts make training more efficient and evaluation easier to improve.