liveBook · Manning

Training dataset

As explained in Causal Inference for Data Science

A training dataset is the subset of observations used to fit a predictive or statistical model.

A training dataset is the portion of observations used to fit a model. Set aside a separate test dataset, fit on the training data, and compare predictions with known outcomes to estimate performance on new data. section 4.1

Choose the training window to balance predictive relevance against the amount of data available:

  • Use observations from a stable context so relationships remain predictive. section 11.3.6
  • Move the window’s beginning farther from the event to gain sample size, accepting possible loss of accuracy from context change. section 11.3.6
  • End the window before treatment or event effects appear; anticipation can make even nominally pretreatment observations unsuitable. section 11.3.6
  • Try alternative endpoints and use cross-validation to select the window with the lowest predictive error. section 11.3.6

Movie-release counterfactual

To simulate viewership without a new movie’s release, train on an earlier pretreatment period. If viewers begin catching up on older movies before release, exclude that anticipated-effect period even though the official event has not occurred. section 11.3.6

Table 11.3 Consequences of choosing endpoints farther from time event T. The Direction arrow shows how the timepoints move with respect to T.
P b start
P b end
T
P a start
P a end
← Direction
Training data window
• More sample size
• Potentially less accuracy due to context change
← Direction
Event anticipation
• Smaller sample size
• Potentially more accuracy with less context change
→ Direction
Delayed effects
• Less data to evaluate the impact
• Potentially less accuracy due to context change
→ Direction
Long-term impact evaluation
• More data to evaluate the impact
• Potentially less accuracy due to context change

Compare how moving training and evaluation endpoints changes sample size, contextual accuracy, and the ability to assess delayed or long-term effects. section 11.3.6

How Causal Inference for Data Science covers it

How deep it goes
Explains splitting, temporal endpoint selection, anticipation effects, sample-size tradeoffs, and cross-validation, but does not develop fitting algorithms.
Examples
Uses exam revision and testing, train-test prediction, and movie-viewership data around a release event.
What it assumes
Assumes familiarity with observations and model predictions while introducing train-test splitting through an exam analogy.
What you can do after
Apply it by separating fitting data from evaluation data and selecting temporal boundaries with cross-validation.
How it connects
Connects to supervised learning, cross-validation, test datasets, predictive modeling, counterfactuals, and causal inference.

Also explained in

Compare how 50 books explain training dataset →

Related concepts

This explainer was generated by Manning's knowledge graph from the text of Causal Inference for Data Science. The section links point to the parts of the book it draws on.