liveBook · Manning

Derived data

As explained in Data for All

Derived data is newly created data produced by transforming, analyzing, or modeling other data rather than directly recording the original activity.

A store visit leaves a record of something that happened. A forecast of next month’s visits is different: it is new data created from earlier records. That is derived data—an extension of raw, naturally occurring data produced through analysis, often by algorithms that carry observed patterns forward. section 7.2.3

Forecasting store activity

Store visits follow weekly, seasonal, and holiday patterns. Before a predicted storm in the US Midwest, shoppers may buy more milk and bread. Analysts can combine past visits with information about store hours, population, and other conditions to estimate future visits and sales. Those estimates can then inform forecasts for staffing, stock levels, and promotions. The forecasts are newly created data, not records of visits or purchases that have already occurred. section 7.2.3

A record does not stop being useful after its first analysis. Data can be aggregated, combined with other sources, and extended into further data. Its lineage connects those later products to the records used to make them. Derived data can also feed synthetic data: estimates for proposed stores, for example, can draw on both raw and derived information when no store activity exists yet to measure. section 7.2 section 7.2.4 section 7.7

Why does that connection matter to you? Your visits and purchases can contribute to forecasts that serve purposes beyond the original transaction. Tracing that path helps make the new data—and questions about who benefits from it—visible. section 7.0 section 7.2.3

How Data for All covers it

How deep it goes
Defines derived data and works through store forecasts, but does not detail forecasting algorithms.
Examples
Uses storm-related grocery shopping and store forecasts for visits, sales, staffing, stock, and promotions.
What it assumes
Starts with familiar shopping activity and explains derived data without requiring knowledge of algorithms.
What you can do after
Evaluate/understand it by distinguishing a forecast from the activity recorded to produce it and tracing its lineage.
How it connects
Links raw data, algorithms, forecasting, data lineage, and synthetic data.

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Data for All. The section links point to the parts of the book it draws on.