Preprocessing

As explained in Machine Learning Platform Engineering

Data preparation is the process of cleaning, transforming, organizing, and otherwise making data suitable for model development.

Preprocessing turns raw data into a reliable foundation for model development. Its quality directly affects model reliability and efficacy, so it belongs inside an iterative, automated ML workflow rather than a one-off notebook task. section 7.0 section 1.1.1

A production-oriented preprocessing workflow typically:

• Explores data interactively with Kubeflow notebooks before formalizing repeatable transformations. section 7.0
• Downloads or collects the dataset as a distinct pipeline stage. section 7.3
• Splits data into training, test, and validation datasets. section 7.3
• Packages each operation as a composable, well-defined Kubeflow component. section 7.3
• Revisits preparation when training or evaluation exposes dataset problems. section 1.1.1 section 1.4.1

Object-detection preparation pipeline

An ID-card detection workflow downloads the dataset, divides it into training, test, and validation sets, and connects the two operations into one executable Kubeflow Pipeline. The same preparation pattern supports broader capstone workflows, including movie recommendation. section 7.0 section 7.3

Figure 7.1 The mental map where we’re now focusing on the first step/component of our pipeline—data collection and preparation (3)

Use the pipeline mental map to locate preprocessing at the data collection and preparation stage. section 7.0

How Machine Learning Platform Engineering covers it

How deep it goes

Breaks preparation into downloading, splitting, notebook exploration, and Kubeflow component construction, while deferring broader pipeline stages.

Examples

Uses ID-card detection and movie recommendation projects, with dataset download and training, test, and validation splits.

What it assumes

Assumes foundational MLOps knowledge while introducing Kubeflow notebooks, components, and pipelines through practical projects.

What you can do after

Build it from scratch as composable Kubeflow components assembled into an executable, reproducible preparation pipeline.

How it connects

Connects data collection, model training, model evaluation, validation, monitoring, Kubeflow notebooks, and Kubeflow Pipelines.

liveBookmanning.com

Read Machine Learning Platform Engineering on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Deep Learning with R, Third EditionData preprocessing is the preparation of raw data by reshaping, scaling, encoding, or otherwise converting it into a model-compatible form.
• 

  liveBookmanning.com
  Data Without LabelsData preprocessing, or data preparation, organizes and transforms raw data into a form suitable for analysis or model training.
• 

  liveBookmanning.com
  Software Testing with Generative AIData preprocessing is the preparation and transformation of data into a form suitable for model training or fine-tuning.
• 

  liveBookmanning.com
  Optimization Algorithms: AI techniques for design, planning, and control problemsA preprocessing phase is an initial computation that transforms or augments data before subsequent queries or operations.
• 

  liveBookmanning.com
  Deep Learning with Python, Third EditionThe transformation of raw data into a form suitable for input to a model.
• 

  liveBookmanning.com
  Grokking Machine LearningData preprocessing is the transformation of raw data into a form suitable for computational analysis or model training.
• 

  liveBookmanning.com
  Ensemble Methods for Machine LearningPreprocessing is the transformation of raw data into a representation suitable for machine-learning algorithms.
• 

  liveBookmanning.com
  Distributed Machine Learning PatternsThe transformation and preparation of raw data into a form suitable for downstream computation or model training.
• 

  liveBookmanning.com
  Grokking AI Algorithms, Second EditionData preparation is the process of cleaning, formatting, and transforming raw data into a form suitable for analysis or model development.
• 

  liveBookmanning.com
  Machine Learning System DesignPreprocessing is the transformation and preparation of data before it is supplied to a machine learning model for training or inference.
• 

  liveBookmanning.com
  Generative AI in ActionData preprocessing is the process of cleaning, transforming, and organizing raw data into a form suitable for analysis or model processing.
• 

  liveBookmanning.com
  Starting Data Analytics with Generative AI and PythonData preprocessing is the transformation and preparation of raw data into a form suitable for analysis or modeling.
• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AIData preprocessing is the transformation and preparation of raw data before it is supplied to a machine learning model.
• 

  liveBookmanning.com
  Graph Neural Networks in ActionData preprocessing is the transformation and preparation of data into a form suitable for model training or inference.
• 

  liveBookmanning.com
  Building Reliable AI SystemsThe cleaning and transformation of raw data before it is supplied to a machine-learning process.
• 

  liveBookmanning.com
  Outlier Detection in PythonThe preparation and transformation of raw data into a form suitable for analysis or machine-learning models.
• 

  liveBookmanning.com
  The Art of AI Product DevelopmentPreprocessing is the transformation and preparation of raw data before it is supplied to a machine-learning model.
• 

  liveBookmanning.com
  Deep Learning with PyTorch, Second EditionData preparation performed before model execution, here separating image and prompt encoding from the main model run.
• 

  liveBookmanning.com
  Knowledge Graphs and LLMs in ActionData preparation is the process of selecting, cleaning, transforming, and organizing data for computational use.
• 

  liveBookmanning.com
  Effective Conversational AIData preparation is the process of transforming collected data into a suitable form for testing or model training.
• 

  liveBookmanning.com
  Evolutionary Deep Learning: Genetic algorithms and neural networksThe process of cleaning, transforming, and organizing data for machine-learning analysis or training.

Compare how 22 books explain preprocessing →

Related concepts

• pipelinepart of · A pipeline is an ordered or coordinated sequence of connected processing stages or tasks that passes data, work, or results toward an outcome.
• data cleaninghas part · Data cleaning is the process of detecting and correcting, removing, transforming, or otherwise handling inaccurate, incomplete, inconsistent, duplicate, or unsuitable data.
• standardhas part · A standard is an established rule, specification, or convention that defines a consistent reference for practice, representation, behavior, or evaluation.
• indexinghas part · Indexing is the organization or use of positions, keys, or lookup structures to identify and retrieve selected data or elements.
• machine learning workflowpart of · A machine learning workflow is an organized sequence of activities for formulating, developing, evaluating, deploying, operating, and maintaining a machine learning system.
• categorical encodinghas part · Categorical encoding transforms discrete category values into numerical representations, such as separate binary indicators or vectors with one active position per category.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning Platform Engineering. The section links point to the parts of the book it draws on.