Chapter 13: Timeseries forecasting

Introduction

Introduces timeseries forecasting as predicting the near future from the recent past, where temporal order matters.

• Temporal Order in Forecasting: Explain that temporal order is central to timeseries and that forecasting uses the recent past to predict the near future.
  Why it matters: This frames the temperature forecasting example that follows.

Key terms: timeseries, temporal order, forecasting

13.1 Different kinds of timeseries tasks

This section defines timeseries data and distinguishes forecasting from other tasks involving data streams.

• Timeseries Structure and Dynamics: Explain what makes data a timeseries and why working with it involves understanding how a system changes over time.
  Why it matters: Those changes over time are what make timeseries data distinct from the other data considered so far.
• Forecasting as Next Prediction: Describe forecasting as predicting what happens next in a timeseries and give a practical reason for making such a prediction.
  Why it matters: Forecasts are useful because they give people time to respond to expected demand or plan ahead.
• Other Timeseries Task Types: Distinguish anomaly detection, classification, and event detection as tasks other than forecasting, and explain why anomaly detection is often unsupervised.
  Why it matters: Not every timeseries problem asks what happens next; the intended task determines what the model should identify.

Key terms: timeseries, dynamics, periodic cycles, trends, forecasting, anomaly detection, unsupervised learning, classification, event detection

13.2 A temperature forecasting example

The section introduces a 24-hour temperature forecasting task, examines patterns in the Jena weather data, and explains why its train, validation, and test splits must preserve time order.

• Temperature Forecasting Data: Describe the task of predicting temperature 24 hours ahead from recent weather measurements, and identify the Jena dataset’s temperature values and other weather quantities as the data used for it.
  Why it matters: This establishes what the example predicts and what observations it uses.
• Temperature Cycles Across Timescales: Explain what the temperature plots reveal at yearly and daily scales, and why predictability can differ between a month-scale forecast and a day-scale forecast.
  Why it matters: Patterns that are clear at one timescale may not make a shorter-term forecast easy.
• Chronological Evaluation Splits: Justify splitting the timeseries into the first 50% for training, the next 25% for validation, and the final 25% for testing, rather than reversing their time order.
  Why it matters: A time-ordered split tests the kind of future prediction the model is meant to perform.

Key terms: timeseries, temperature, features, Jena, periodicity, yearly periodicity, daily periodicity, training, validation, test

13.2.1 Preparing the data

The section prepares numerical weather data as normalized, windowed training, validation, and test datasets for predicting temperature 24 hours ahead.

• Independent timeseries normalization: Explain why each weather timeseries is normalized independently and why its mean and standard deviation are computed from training data only.
  Why it matters: It makes differently scaled weather measurements suitable as inputs to the forecasting model.
• Forecast windows and targets: Describe how timeseries_dataset_from_array extracts input sequences and how its targets must align with those sequences for forecasting.
  Why it matters: Correct target alignment determines whether a window is paired with the future value the model is meant to predict.
• On-the-fly sample generation: Justify generating forecasting samples on the fly instead of storing every input window.
  Why it matters: It explains why the dataset utility is useful beyond merely creating windows.
• Forecast dataset configuration: Describe how sampling, sequence length, target delay, and time-based dataset boundaries implement the stated forecasting task.
  Why it matters: These settings turn the weather record into examples that match the precise five-day-input, 24-hour-ahead prediction problem.

Key terms: normalize, mean, standard deviation, training data, timeseries_dataset_from_array, sequences, targets, sequence_length, on the fly, raw_data

13.2.2 A commonsense, non-machine-learning baseline

The section establishes and evaluates a commonsense temperature forecast that later models must beat.

• Purpose of a commonsense baseline: Explain why a simple baseline should be evaluated before a machine-learning model: it provides a sanity check and a score the model must beat to demonstrate usefulness.
  Why it matters: A model’s score means more when you compare it with what a simple rule can already achieve.
• Temperature persistence forecast: Describe the section’s temperature baseline, why it is plausible, and how well it performs: predict the temperature 24 hours ahead to equal the current temperature, then compare predictions with actual temperatures using MAE.
  Why it matters: This concrete error gives the later temperature models a target to beat.

Key terms: commonsense baseline, sanity check, temperature timeseries, daily period, mean absolute error (MAE), validation MAE, test MAE

13.2.3 Let’s try a basic machine learning model

This section tests a small densely connected temperature-forecasting model against a commonsense baseline and explains why a trainable model may fail to find a simple, effective solution.

• Simple model before complexity: Justify trying a small, cheap machine-learning model before a more complex model, and describe what its validation results show relative to the commonsense baseline.
  Why it matters: A more complex model should earn its complexity by improving on simpler alternatives.
• Dense regression model setup: Describe the model's basic structure and explain the choices of output activation, training loss, and monitored metric.
  Why it matters: These choices distinguish what the model optimizes from the error measure reported for comparison.
• Limits of hypothesis search: Explain why the trained network may fail to discover the effective commonsense heuristic even if its hypothesis space can represent it, and state the implication for model design.
  Why it matters: Model design can guide learning toward a simple solution that an unrestricted search might miss.

Key terms: commonsense baseline, no-learning baseline, validation losses, densely connected networks, flattening, Dense layers, regression, mean squared error (MSE), MAE, compile()

13.2.4 Let’s try a 1D convolutional model

The section tries a 1D ConvNet for temperature forecasting and explains why its assumptions make it perform poorly on weather data.

• Temporal convolution and invariance: Explain how a 1D ConvNet applies sliding windows to a sequence and reuses representations at different positions, and why this suits sequences whose window contents have similar properties regardless of position.
  Why it matters: This assumption explains both why the architecture seemed promising and why its fit to weather data needs scrutiny.
• Temperature model window choice: Describe the window choice in the temperature-forecasting example and what happens to the window size as the model downsamples the sequence.
  Why it matters: The window choice connects the model design to the daily cycle observed in the input.
• ConvNet forecasting limitations: Explain why this 1D ConvNet performed worse than the dense model and the commonsense baseline, addressing both time-of-day differences and the importance of recent observations.
  Why it matters: The disappointing result shows why an architecture's assumptions must match the time-dependent structure of the data.

Key terms: temporal ConvNet, Conv1D, translation invariance assumption, window length, MaxPooling1D, daily cycles, validation MAE, translation invariance, order information, max pooling

13.3 Recurrent neural networks

This section explains why preserving sequence order matters for temperature forecasting and reports that an LSTM model modestly outperforms a commonsense baseline.

• Timeseries Order and Causality: Explain why the earlier densely connected and convolutional approaches were poor fits for this forecasting task, and why the data should instead be treated as a sequence.
  Why it matters: Preserving time order helps explain the choice of a sequence-oriented model for this task.
• LSTM Forecasting Result: State how the LSTM-based model performed on the temperature forecasting task relative to the commonsense baseline.
  Why it matters: The result shows that a machine-learning model can improve on the simple baseline for this task.

Key terms: timeseries, flattened, pooling, sequence, causality, order, LSTM-based model, validation MAE, test MAE, commonsense baseline

13.3.1 Understanding recurrent neural networks

This section contrasts memoryless feedforward networks with RNNs and explains a simple RNN’s stateful forward pass and outputs.

• Feedforward and recurrent processing: Distinguish how feedforward networks and RNNs process a sequence. Explain that a feedforward network processes inputs independently, whereas an RNN iterates through sequence elements while retaining information from earlier elements.
  Why it matters: This distinction explains why an RNN is suited to processing ordered data one element at a time.
• State across independent sequences: Explain why an RNN’s state is reset between independent sequences, even though it persists across elements within one sequence.
  Why it matters: The reset marks the boundary between using earlier steps as context and mixing information from unrelated samples.
• Simple RNN forward pass: Trace the toy RNN’s forward pass: start with a zero state, combine the current input and state to produce an output at each timestep, then use that output as the next state. Describe the step transformation as parameterized by two matrices and a bias vector.
  Why it matters: Following one step of the loop makes the RNN’s otherwise abstract notion of memory concrete.
• Sequence outputs and final output: Explain what the toy RNN’s output at a timestep represents and why its final output can sometimes replace the full output sequence.
  Why it matters: It explains when a model might use the output at every timestep and when the final output alone may suffice.

Key terms: feedforward networks, recurrent neural network (RNN), state, internal loop, independent sequences, single data point, forward pass, timesteps, initial state, step function

13.3.2 A recurrent layer in Keras

This section shows how Keras recurrent layers handle sequence inputs and outputs, then explains why LSTM uses a carry track instead of relying on SimpleRNN for long-term information.

• Recurrent layer input shapes: Describe the shape of a SimpleRNN input batch and how the Input shape can allow variable-length sequences.
  Why it matters: The input shape determines whether the layer can accept sequences of varying lengths.
• Recurrent output modes and stacking: Distinguish returning the last output from returning the full output sequence, and explain which mode intermediate recurrent layers need when stacked.
  Why it matters: Choosing the output mode determines whether a recurrent layer can pass a sequence to the next layer.
• LSTM carry and long-term information: Explain why SimpleRNN struggles with long-term dependencies and how the LSTM carry track is meant to help.
  Why it matters: The carry track explains the practical motivation for using LSTM rather than a plain SimpleRNN.
• Learned roles of LSTM operations: Distinguish an intuitive interpretation of LSTM's carry updates from what the architecture alone actually determines.
  Why it matters: It prevents treating the names of cell operations as guarantees about the behavior a trained LSTM learns.

Key terms: SimpleRNN, batch_size, timesteps, input_features, Input(), return_sequences, full sequences, last output, rank-3 tensor, rank-2 tensor

13.3.3 Getting the most out of recurrent neural networks

Previews three advanced RNN techniques that will be used to refine the temperature forecasting model.

13.3.4 Using recurrent dropout to fight overfitting

Explains how temporally consistent dropout regularizes recurrent networks, then shows its effect and runtime tradeoffs in a temperature-forecasting LSTM.

• Temporally consistent dropout masks: Explain why dropout in a recurrent network uses the same mask at every timestep and how recurrent dropout also regularizes the layer’s internal recurrent activations.
  Why it matters: The timing of the mask determines whether dropout helps regularize the RNN or interferes with its learning.
• LSTM recurrent dropout results: Describe how the example adds recurrent dropout to the LSTM and what the training and evaluation results show.
  Why it matters: The example connects the regularization technique to observable improvements in the forecasting model.
• Recurrent dropout GPU tradeoff: Explain why enabling recurrent dropout can slow a Keras LSTM or GRU on a GPU despite unchanged computational cost.
  Why it matters: A useful regularization setting can change which GPU implementation runs and therefore change training speed.
• RNN unrolling constraints: Describe how unrolling may speed an RNN that cannot use cuDNN and state the conditions that limit its use.
  Why it matters: Unrolling offers a possible speedup when cuDNN is unavailable, but its memory and input-shape requirements restrict when it is practical.

Key terms: dropout mask, timestep, recurrent dropout mask, recurrent activations, dropout, recurrent_dropout, LSTM, validation MAE, test MAE, cuDNN kernel

13.3.5 Stacking recurrent layers

This section explains when stacking recurrent layers can increase model capacity, how Keras passes sequences between those layers, and what the stacked GRU achieved in the temperature-forecasting example.

• Capacity after a performance bottleneck: Explain why increasing recurrent-network capacity is worth trying when performance has stalled but overfitting is no longer the main problem.
  Why it matters: It gives you a reason to try a larger model rather than adding complexity by default.
• Full sequences between recurrent layers: Describe what intermediate Keras recurrent layers must return when they are stacked and how that is specified.
  Why it matters: Passing the full sequence lets the next recurrent layer operate on the sequence, not just its final output.
• Modest gains from stacking: Describe the reported result of the two-layer dropout-regularized GRU model and what the section concludes about adding capacity.
  Why it matters: The example shows that more capacity can help without guaranteeing a large improvement.

Key terms: capacity, expressive power, overfitting, dropout, recurrent layer stacking, intermediate layers, full sequence of outputs, rank-3 tensor, last timestep, return_sequences=True

13.3.6 Using bidirectional RNNs

This section explains how bidirectional RNNs combine forward and reversed sequence processing, and why that helps some tasks but not the temperature-forecasting example.

• Sequence order and task: Explain why reversing input sequences changes an RNN’s representations, and contrast the effect on temperature forecasting with the effect on text.
  Why it matters: The usefulness of each processing direction depends on what the task’s sequence order means.
• Bidirectional RNN representations: Describe how a bidirectional RNN uses two recurrent networks and why combining their representations can capture more of a sequence.
  Why it matters: The two directions can provide complementary views of the same sequence.
• Bidirectional layer task fit: Describe how Keras constructs a bidirectional recurrent layer and explain why it underperforms a plain LSTM on this temperature task despite being well suited to some text tasks.
  Why it matters: Adding a processing direction helps only when that direction supplies useful information for the task.

Key terms: order dependent, chronological order, antichronological order, commonsense baseline, bidirectional RNN, chronological, antichronological, representations, Bidirectional layer, plain LSTM layer

13.4 Going even further

The section suggests ways to improve the temperature forecaster through empirical iteration, then explains limits imposed by the available weather data and cautions against applying the same approach to financial markets.

• Empirical forecasting improvements: Describe model and input changes worth trying on the temperature forecasting problem, and explain why their value must be tested rather than assumed.
  Why it matters: It keeps model improvement grounded in measured results rather than presumed best settings.
• Single location weather limits: Explain why the temperature forecaster may improve only modestly over the no-learning baseline when its measurements come from one location.
  Why it matters: It explains why further model tuning may not overcome missing information in the inputs.
• Market forecasting caveat: Distinguish forecasting settings where the past can predict the future from financial markets, and explain why public data and familiar machine-learning methods offer little trading advantage.
  Why it matters: It prevents an unjustified leap from a workable weather forecast to a profitable trading strategy.

Key terms: recurrent layer, dropout, learning rate, Adam optimizer, sampling rate, feature engineering, no-learning baseline, single location, wide grid of different locations, past performance

13.5 Summary