Time-series validation

As explained in Machine Learning System Design

A validation procedure that separates training and evaluation data according to their temporal order.

Time-series validation evaluates models when observations depend on time. Random splits can let future data inform predictions about the past, so training data must precede test data. section 7.2.4

When choosing a validation scheme:

• Set the test window to reflect the forecast horizon and how long the model will run before retraining. section 7.2.4
• Choose whether to use all available history or limit training to one or two recent periods. section 7.2.4
• Match training and test sizes to seasonal cycles: yearly patterns call for at least two years of training, while weekly test folds should contain the same weekdays. section 7.2.4
• Consider a gap to account for delays in receiving data and features, and to reduce correlation and leakage risk. section 7.2.4

Figure 7.4 Standard time-based split. The test dataset always follows the train one, so train samples are “past” and test is “future.”

The split places training observations in the past and test observations in the future. section 7.2.4

How Machine Learning System Design covers it

How deep it goes

Specifies timestamp-ordered splitting and choices for test windows, training history, seasonality, and gaps.

Examples

Uses neighboring-day sales, later user actions, yearly patterns, and whole-week test folds as examples.

What you can do after

Evaluate models with timestamp-ordered splits sized for the forecast horizon, retraining cadence, and seasonal patterns.

How it connects

Relates time-series validation to holdout sets, cross-validation, and ordinal K-fold.

liveBookmanning.com

Read Machine Learning System Design on liveBook Get the book at manning.com

Also explained in

• 

  liveBookmanning.com
  Machine Learning for Tabular Data: XGBoost, Deep Learning, and AITime-based validation evaluates a model using temporally ordered training and validation data that respect the direction of time.

Compare how 2 books explain time-series validation →

Related concepts

• cross-validationcontrasts with · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.
• validationkind of · The process of checking data, behavior, systems, models, or results against defined criteria to assess their correctness, suitability, or acceptability.
• random forestmeasures · A random forest is an ensemble model that combines predictions from multiple decision trees built with randomized data samples, feature selections, or both.
• seasonality effectconfigured by · A seasonality effect is a recurring pattern of variation in data associated with regular temporal, calendar, or time-of-year cycles.

This explainer was generated by Manning's knowledge graph from the text of Machine Learning System Design. The section links point to the parts of the book it draws on.