liveBook · Manning

Data cleaning

As explained in Demand Forecasting Best Practices

Data cleaning is the detection and correction of errors, inconsistencies, missing values, and unsuitable representations in datasets.

Data cleaning prepares demand and demand-driver data for forecasting by finding inaccurate, inconsistent, missing, or unsuitable information and correcting its underlying causes. In demand planning, bad data can defeat a good model, so cleaning is part of building a dependable forecasting process rather than a final cosmetic step. section 14.4.1 section 14.4.2

Use data cleaning as an operational investigation:

  • Define the forecasting objective and aggregation first, then gather demand data rather than relying on sales data alone. section 14.4.1
  • Assess whether collecting and cleaning additional demand drivers—such as prices, promotions, weather, or other external information—is worth its expected forecasting benefit. section 14.2.2 section 14.4.1
  • When errors appear, inspect the root cause instead of manually smoothing outliers; inaccurate data requires correction, while missing drivers require more relevant data. section 14.4.2
  • Use error graphs or dashboards across training and test data to create a feedback loop for finding overlooked information and improving the dataset. section 14.4.2

A forecast error with a hidden cause

A historical price increase reduces demand, but the model has no price data. The error is not fixed by manually smoothing the affected observations: collect and clean price information, then ensure the model can use that demand driver. Similar checks can reveal promotions, holidays, or shortages that explain unusually large errors. section 14.4.2

Figure 14.12 4-step model creation loop

Use the model-creation loop to connect error inspection with root-cause analysis, data cleaning, and subsequent model improvement. section 14.4.2

How Demand Forecasting Best Practices covers it

How deep it goes
Covers collection, root-cause correction, and error feedback, but does not prescribe specific cleaning algorithms.
Examples
Uses Excel files, prices, promotions, holidays, shortages, weather, and external market data as cleaning cases.
What it assumes
Assumes a defined forecasting objective and aggregation while distinguishing demand data from sales data.
What you can do after
Diagnose forecast errors by tracing them to inaccurate data or missing demand drivers, then collect and clean relevant data.
How it connects
Connects demand data, demand drivers, demand forecasting, model errors, overfitting, and underfitting.

Also explained in

Compare how 14 books explain data cleaning →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Demand Forecasting Best Practices. The section links point to the parts of the book it draws on.