Chapter 8: Model training and validation: Part 1

Introduction

This chapter introduction shifts the pipeline from data preparation to reproducible model training and validation, using object detection as the main example.

• Reliable training and validation: Explain why an accurate model alone is not enough for a reliable ML system: training should be reproducible and validation should be robust.
  Why it matters: Reliable workflows make model results more useful beyond a one-off experiment.
• Pipeline component roadmap: Describe the chapter’s move from the data preparation pipeline to modular training and validation components, using YOLO for ID card detection as the hands-on example and integrating the components into an ML pipeline.
  Why it matters: The roadmap connects prepared data to the next two pipeline steps: training and evaluation.
• Tracking and data access themes: State that the chapter also introduces capturing metrics and artifacts in tracking frameworks and using different ways to access training and evaluation data; note that these workflow practices will later be applied to a movie recommendation system.
  Why it matters: These themes show that the workflow must handle results and data access, not just run training code.

Key terms: reproducible training processes, robust validation strategies, reliable ML systems, modular training components, validation components, You Only Look Once (YOLO), ID card detection system, ML pipelines, metrics and artifacts, tracking frameworks

8.1 Training an object detection model

This section introduces a component-based approach to training the ID card detection model with YOLO, explaining why it suits production pipelines and what the training component will do.

• Production training components: Explain why the section designs training as pipeline components rather than relying on a Jupyter Notebook: components can be versioned, run consistently across environments, integrated with data preparation, and monitored and evaluated.
  Why it matters: These properties make training easier to repeat and manage in a production ML system.
• YOLO training component responsibilities: Describe the three planned responsibilities of the ID card detection training component: download images and labels for the training, validation, and test splits; configure the dataset paths and label correspondence; and train and evaluate the model. State that the Ultralytics YOLO library handles much of the training and evaluation work, while the code remains customizable.
  Why it matters: This outline helps you follow what each part of the upcoming training implementation is for.

Key terms: reproducible, modular, pipeline infrastructure, source control, YOLO, Ultralytics YOLO, training component, training, validation, and test images, labels

8.1.1 Training YOLO on a custom dataset

This section prepares a custom ID-card dataset for YOLOv8 training by defining its data configuration and writing it to YAML.

• Custom dataset configuration: Describe how to define a data-configuration dictionary for the ID-card dataset and write it to a YAML file before training. State that it identifies the image paths and lists `id_card` as the sole class, with more classes added if other objects are to be detected.
  Why it matters: YOLOv8 needs the data configuration to find the images and learn which objects to detect.
• Stage-specific label locations: Explain where YOLOv8 expects labels relative to the images for each dataset stage, using `train/labels` as the training-set example. Distinguish this directory convention from the image paths specified in the configuration.
  Why it matters: The directory layout lets YOLOv8 find the corresponding labels without separate label paths in the configuration.

Key terms: YOLOv8, data configuration, YAML, id_card, labels, images, train/labels

8.1.2 Training the model

This section explains how to configure YOLOv8 training for ID card detection, choose a model size, and retain outputs for evaluation.

• Production training configuration: Explain why the training component uses robust defaults rather than manually tuning every hyperparameter, while exposing settings according to the project’s needs. Describe the balance among training efficiency, compute resources, and accuracy requirements.
  Why it matters: That balance helps make training practical and repeatable outside an experiment.
• YOLO training inputs and outputs: Describe how the component initializes a pretrained model with Ultralytics’ YOLO class and calls `train`, including the roles of the dataset configuration, image size, epochs, batch size, name, and project path. Explain that the project path collects weights and evaluation artifacts, including automatically captured inference samples.
  Why it matters: These inputs and outputs connect the training run to the pipeline and later validation.
• YOLO model size choice: Explain the speed, memory, compute, and accuracy trade-off between smaller and larger YOLOv8 models. Justify starting the ID card detector with YOLOv8 Nano as a low-cost feasibility test that can inform a later move to a larger model.
  Why it matters: The model size determines whether the system can meet its accuracy needs within its resource budget.

Key terms: robust default configurations, hyperparameters, training component, Ultralytics, YOLO class, train method, data, imgsz, epochs, YOLOv8n

8.1.3 Container components for system dependencies

This section builds a YOLO training step as a Kubeflow container component so it can install system dependencies, run a training script, and pass typed artifacts through a pipeline.

• System dependency workaround: Explain why YOLO training may need a container component rather than a standard Python-package component: the Ultralytics image can lack libGL.so.1, and fixing that requires installing libgl1-mesa-glx with apt-get.
  Why it matters: This explains when the extra complexity of a container component is worthwhile.
• Standalone YOLO training script: Describe how the independently executable script parses training arguments, sets up YOLO's data configuration, trains for the specified epochs, and saves the best weights to the model-output path.
  Why it matters: Keeping training logic in a runnable script makes clear what the container component must invoke and supply.
• Typed container component interface: Describe how @dsl.container_component and dsl.ContainerSpec specify the Docker image and commands while exposing epochs, batch, model name, three Dataset inputs, and model and YAML outputs as KFP artifacts.
  Why it matters: The typed interface connects a customizable container to the rest of the Kubeflow pipeline.
• Script injection and argument bridge: Trace how the container specification writes the training script with a heredoc, executes it, and maps KFP inputs and artifact paths to command-line arguments through Bash positional parameters; state that the resulting component is used in a pipeline like a standard component.
  Why it matters: This is the bridge between KFP's typed artifacts and the independently runnable YOLO script.

Key terms: libGL.so.1, libgl1-mesa-glx, apt-get, @dsl.component, @dsl.container_component, parse_args, data.yaml, epochs, best model, model_output

8.1.4 Creating the validation component

This section builds a Kubeflow component that validates a trained YOLO model and records its performance as pipeline metrics.

• Production validation purposes: Explain why production model validation goes beyond measuring accuracy: it can reveal degradation and differences across data segments, support deployment decisions, and preserve performance history.
  Why it matters: Validation results help teams decide whether a model is ready to deploy and where it needs improvement.
• Validation component artifact flow: Describe the KFP validation component’s four artifact types: a generic Artifact for configuration, a Model for trained weights, a Dataset for validation data, and Metrics for evaluation results. Explain that these types let KFP manage data flow and dependencies between components.
  Why it matters: Typed artifacts make it clear what the validation step consumes and what it passes on to the pipeline.
• YOLO validation metric logging: Trace how the component loads trained YOLO weights, validates the model against a test dataset with model.val(), and records standard COCO mean Average Precision (mAP) values through KFP’s metrics-logging interface.
  Why it matters: Logging validation scores as pipeline outputs makes model performance available for review and comparison.
• Validation component dependency setup: Explain why the validation step uses dsl.component with the Ultralytics Docker image as base_image and specifies remaining dependencies through packages_to_install, rather than installing every ML dependency through requirements.txt.
  Why it matters: A prepackaged image reduces setup work and the risk of conflicting ML library versions.

Key terms: validation, model degradation, data segments, historical performance records, Artifact, Model, Dataset, Metrics, KFP, model.val()

8.1.5 Creating the pipeline

This subsection adds a validation component to the existing YOLO training pipeline and points to compiling, uploading, and executing it next.

• Training and validation pipeline: Describe how the pipeline shown here differs from the earlier version: it now includes the validation component alongside the existing training workflow. State that compiling, uploading, and executing the pipeline come next.
  Why it matters: Adding validation makes the pipeline ready for the execution and artifact checks that follow.

Key terms: validation component, pipeline, compile, upload, execute

8.1.6 Executing the pipeline

The section shows how to run the object-detection pipeline in Kubeflow, interpret its execution, and find its metrics and output artifacts.

• Pipeline run and dependencies: Explain that a KFP run starts with parameters prepopulated from the pipeline defaults, and that connections in the topology represent data dependencies between components, including an output from split-dataset used by validate-model.
  Why it matters: The run graph shows why components execute in a particular relationship.
• Training output and runtime: Describe the information Ultralytics reports while training and explain why checking the time per epoch is useful; the example dataset took about an hour for one epoch on the authors’ cluster, though runtime varies.
  Why it matters: Training output helps you monitor progress and estimate how long a run may take.
• Run metrics and artifacts: Describe where to inspect a successful run’s metrics: the Run output tab, their raw form under Output artifacts, and the Pipelines Overview page, where they appear alongside previous runs. State that the training artifacts include model weights that the next section will download for local use or later deployment.
  Why it matters: Knowing where metrics and artifacts appear lets you review a run and find its trained model.

Key terms: KFP run, run parameters, pipeline topology, data dependencies, split-dataset, validate-model, Ultralytics, epoch, model architecture, GPU memory

8.1.7 Validating model artifacts

The section shows how to inspect YOLO object-detection artifacts manually, compare pretrained and trained predictions, and turn those observations into validation checks and training decisions.

• Manual artifact verification: Explain why manual inference complements automated pipeline validation, and describe testing sample images first with the default YOLOv8 Nano model and then with the trained model's best.pt weights downloaded from the pipeline's project output artifact.
  Why it matters: Inspecting actual predictions can expose failures that a successful pipeline run alone does not reveal.
• Detection quality signals: Describe how to judge detections from their class labels, confidence scores, and bounding boxes, using the section's contrast between the generic model and the ID-card-trained model.
  Why it matters: A detection is useful only if its label, certainty, and location make sense together.
• Epochs and selected weights: Explain how additional training improved the example ID-card detections and distinguish the number of epochs requested from the epoch whose validation performance produced best.pt.
  Why it matters: The saved best weights are only as good as the training opportunity and validation results behind them.
• Automated detection checks: Describe how observations from manual inspection can become programmatic validation: check detection confidence against a threshold and bounding-box coordinates against reasonable values with a tolerance.
  Why it matters: Repeatable checks help catch implausible predictions without requiring someone to inspect every image.
• Iterative hyperparameter tuning: Explain the section's small-start approach to tuning: begin with few hyperparameters, then adjust learning rate, batch size, or epoch count using results from earlier training jobs, especially if a model fails to converge.
  Why it matters: Changing settings in response to observed results makes training experiments more purposeful.

Key terms: manual inspection, default YOLO model, YOLOv8 Nano, Output artifacts, project, best.pt, confidence score, bounding box, class label, ID card

8.2 Summary

The section summarizes how to choose a suitable model, deliver training data to automated pipelines, and make training results easy to inspect and validate.