Preprocessing turns collected, heterogeneous data into reliable model inputs. In a retail forecasting system, it follows existing ETL and quality checks rather than replacing them: transactions, stock, promotions, product and store metadata, weather, and competitor prices must be made usable together. section 6.7.1
A practical preprocessing pass can include:
- Filter missing or duplicated transaction, stock, and promotion records before aggregation. section 6.7.1
- Aggregate transactions daily, optionally rewriting the previous two or three days to repair corrupt or incomplete partitions. section 6.7.1
- Join internal and external sources using date, product ID, or store ID, then calculate features from the joined data. section 6.7.1
- Apply modality-specific transformations such as feature scaling, one-hot encoding, image normalization and resizing, or text tokenization. section 10.1.1
Treat preprocessing as part of a larger directed acyclic graph. A training pipeline fetches data, preprocesses it, trains and evaluates a model, then packages artifacts; an inference pipeline accepts raw data and produces predictions. Keeping these roles distinct helps clarify where transformations belong. section 10.1.1
Retail forecasting
Competitor prices cover only about a quarter of the product catalog and contain gaps, while other sources may contain duplicates or missing values. Preprocessing therefore combines quality checks, daily aggregation, repair of recent partitions, source joins, and feature calculation before forecasting. section 6.7.1
Figure 10.1 A DAG scheme representing a training pipeline
A DAG view makes preprocessing visible as one step in the training pipeline, alongside fetching, training, evaluation, postprocessing, reporting, and artifact packaging. section 10.1.1
