Conformal prediction

As explained in Time Series Forecasting Using Foundation Models

Conformal prediction is a model-agnostic, distribution-free method for constructing prediction intervals with a specified coverage level.

Conformal prediction is a nonparametric, model-agnostic framework for quantifying forecast uncertainty without assuming a fixed statistical distribution. A user-specified confidence level controls the intended proportion of actual values inside the prediction interval. section 3.2.2

The method uses historical predictions and actual values to calculate conformity scores, such as absolute errors. It adds and subtracts these scores from each point forecast, then uses quantiles to form intervals. Its central assumption is exchangeability; die rolls illustrate this assumption, whereas time-series order does not satisfy it directly. section 3.2.2

Figure 3.4 Overview of conformal prediction for time-series data. Historical predictions are used to calculate conformity scores, which are used to generate a range of possible values for each point forecast. From that range of possible values, we calculate the quantiles corresponding to a user-specified confidence level.

Trace the time-series workflow from historical predictions and conformity scores to quantiles and prediction intervals. section 3.2.2

A practical anomaly-detection workflow:

• Configure PredictionIntervals with a confidence level, run cross-validation, and refit the forecasting model at every window. section 9.5 section 9.5.1
• Flag an observation as anomalous when it falls below the lower interval or above the upper interval. section 9.5.1
• Treat interval calibration and computational cost as practical trade-offs: short training produced many false anomalies and low precision. section 9.5.1

Website visits and taxi rides

For a forecast of 300 daily website visits, an 80% interval might span 200–400, while a 90% interval might span 100–500. In anomaly detection, conformal intervals applied to daily New York City taxi rides identify values outside a 99% interval, though poorly calibrated intervals can overflag normal observations. section 3.2.2 section 9.5.1

How Time Series Forecasting Using Foundation Models covers it

How deep it goes

Defines the framework and works through conformity scores and quantiles, but stops short of a full theoretical treatment.

Examples

Uses website visits, die rolls, historical forecasts, and daily taxi rides to illustrate intervals and anomaly flags.

What it assumes

Builds from point forecasts and cross-validation while defining exchangeability through a die-rolling example.

What you can do after

Apply it via neuralforecast to estimate intervals through cross-validation and flag observations outside selected bounds.

How it connects

Connects prediction intervals with time-series forecasting, cross-validation, anomaly detection, and exchangeability.

liveBookmanning.com

Read Time Series Forecasting Using Foundation Models on liveBook Get the book at manning.com

Related concepts

• TimeGPTused by · TimeGPT is a proprietary foundation model designed for time-series forecasting and related time-series analysis.
• prediction intervalproduces · A prediction interval is a range computed from a forecasting model that is expected to contain a future or unobserved value at a specified confidence level.
• cross-validationused by · Cross-validation is a resampling technique that evaluates a model by repeatedly fitting it on some portions of available data and assessing it on held-out portions.

This explainer was generated by Manning's knowledge graph from the text of Time Series Forecasting Using Foundation Models. The section links point to the parts of the book it draws on.