> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Forecasting Time Series with RNNs

> Baselines, simple and deep RNNs, LSTM and GRU in PyTorch.

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/rnn/time-series-forecasting/index.ipynb)

In this section you train recurrent networks to forecast a time series. You start from two baselines that any useful model must beat, build the cell of the [simple RNN section](https://aegean.ai/aiml-common/lectures/rnn/simple-rnn/index.md) by hand, and then train deeper RNNs, an [LSTM](https://aegean.ai/aiml-common/lectures/rnn/lstm/index.md) and a GRU on the same data. The models predict one step ahead first, and then ten steps ahead.

The examples follow chapter 15 of Aurélien Géron's [Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow](https://github.com/ageron/handson-ml2) (2nd edition), rewritten for PyTorch.

> [accordion] Code
>
> ```python theme={null}
> import os
> import random
>
> os.environ.setdefault("CUBLAS_WORKSPACE_CONFIG", ":4096:8")
>
> import numpy as np
> import torch
> import torch.nn as nn
> import torch.nn.functional as F
> from torch.utils.data import DataLoader, TensorDataset
>
> import matplotlib as mpl
> import matplotlib.pyplot as plt
> from cycler import cycler
> from matplotlib_inline.backend_inline import set_matplotlib_formats
> set_matplotlib_formats("svg")
> FIG_GRAY = "#808080"
> FIG_COLORS = ["#0072B2", "#D55E00", "#009E73", "#cb77a6", "#c28600", "#1d99e0", "#7f7f7f"]
> plt.rcParams.update({
>     "figure.facecolor": "none", "axes.facecolor": "none",
>     "savefig.facecolor": "none", "savefig.transparent": True,
>     "text.color": FIG_GRAY, "axes.labelcolor": FIG_GRAY, "axes.edgecolor": FIG_GRAY,
>     "xtick.color": FIG_GRAY, "ytick.color": FIG_GRAY,
>     "grid.color": FIG_GRAY, "grid.alpha": 0.25,
>     "legend.facecolor": "none", "legend.edgecolor": FIG_GRAY, "legend.framealpha": 0,
>     "axes.prop_cycle": cycler(color=FIG_COLORS),
>     "svg.hashsalt": "aiml", "svg.fonttype": "path",
> })
>
> device = torch.device("cuda" if torch.cuda.is_available() else
>                       "mps" if torch.backends.mps.is_available() else "cpu")
> torch.backends.cudnn.deterministic = True
> torch.backends.cudnn.benchmark = False
> torch.use_deterministic_algorithms(True, warn_only=True)
>
>
> def set_seed(seed=42):
>     """Seed Python, NumPy and PyTorch so every run gives the same numbers."""
>     random.seed(seed)
>     np.random.seed(seed)
>     torch.manual_seed(seed)
>
>
> print(f"Device: {device}")
> ```

```output theme={null}
Device: cuda
```

## The dataset

Each example is a sum of two sine waves with random frequencies and phases, plus a little uniform noise. All series have the same length and live roughly in $[-1, 1]$:

$$
\begin{aligned}
x(t) = {} & 0.5 \sin\big((t - o_1)(10 + 10 f_1)\big) \\
& + 0.2 \sin\big((t - o_2)(20 + 20 f_2)\big) + \epsilon_t
\end{aligned}
$$

where $f_1, f_2, o_1, o_2$ are drawn uniformly from $[0, 1)$ once per series, and $\epsilon_t$ is drawn uniformly from $[-0.05, 0.05)$ at every step. Each series is sampled at equally spaced times $t$ in $[0, 1]$.

The input is the first 50 values of a series and the target is value 51. The 10,000 series are split into 7,000 for training, 2,000 for validation and 1,000 for testing. The input tensor has shape (batch, time steps, features), here with a single feature per step.

> [accordion] Code
>
> ```python theme={null}
> def generate_time_series(batch_size, n_steps):
>     freq1, freq2, offsets1, offsets2 = np.random.rand(4, batch_size, 1)
>     time = np.linspace(0, 1, n_steps)
>     series = 0.5 * np.sin((time - offsets1) * (freq1 * 10 + 10))   # wave 1
>     series += 0.2 * np.sin((time - offsets2) * (freq2 * 20 + 20))  # wave 2
>     series += 0.1 * (np.random.rand(batch_size, n_steps) - 0.5)    # noise
>     return series[..., np.newaxis].astype(np.float32)
>
>
> np.random.seed(42)
> n_steps = 50
> series = generate_time_series(10000, n_steps + 1)
> X_train, y_train = series[:7000, :n_steps], series[:7000, -1]
> X_valid, y_valid = series[7000:9000, :n_steps], series[7000:9000, -1]
> X_test, y_test = series[9000:, :n_steps], series[9000:, -1]
> print("X_train:", X_train.shape, " y_train:", y_train.shape)
> ```

```output theme={null}
X_train: (7000, 50, 1)  y_train: (7000, 1)
```

Three validation series and the value that follows each of them. The gray dots are the 50 inputs and the blue dot is the target. In the figures that follow, blue marks targets and orange crosses mark predictions.

![Output from cell 4](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_4_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=c7438611f963729cbf963cb2ce6770fb)

## Baselines

A forecaster is only useful if it beats simple rules. The *naive* forecast predicts that the next value equals the last observed value, $\hat x(51) = x(50)$. All errors in this section are mean squared errors (MSE) on the validation set.

> [accordion] Code
>
> ```python theme={null}
> def mse(y_true, y_pred):
>     return float(np.mean((y_true - y_pred) ** 2))
>
>
> results = {}
> y_pred = X_valid[:, -1]
> results["Naive, 1 step"] = mse(y_valid, y_pred)
> print(f"Naive forecast MSE: {results['Naive, 1 step']:.4f}")
> ```

```output theme={null}
Naive forecast MSE: 0.0202
```

![Output from cell 6](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_6_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=8ae98c7977bc87306efa18dd8999592c)

The naive forecast has a validation MSE of 0.0202.

The second baseline is a linear model. It flattens the 50 inputs into a vector and predicts the next value as a weighted sum, $\hat x(51) = \mathbf w^\top \mathbf x + b$, with 51 parameters.

Every model in this section trains the same way: Adam on the MSE loss, mini-batches of 32 series, 20 epochs.

> [accordion] Code
>
> ```python theme={null}
> def to_tensor(a):
>     return torch.from_numpy(np.ascontiguousarray(a, dtype=np.float32))
>
>
> @torch.no_grad()
> def predict(model, X, batch_size=1000):
>     model.eval()
>     X = to_tensor(X)
>     out = [model(X[i:i + batch_size].to(device)).cpu() for i in range(0, len(X), batch_size)]
>     return torch.cat(out).numpy()
>
>
> def fit(model, X_tr, Y_tr, X_va, Y_va, epochs=20, lr=1e-3, batch_size=32, seed=42):
>     """Train with Adam on the MSE loss. Returns per-epoch training and validation loss."""
>     model.to(device)
>     optimizer = torch.optim.Adam(model.parameters(), lr=lr)
>     loader = DataLoader(TensorDataset(to_tensor(X_tr), to_tensor(Y_tr)), batch_size=batch_size,
>                         shuffle=True, generator=torch.Generator().manual_seed(seed))
>     history = {"loss": [], "val_loss": []}
>     for epoch in range(epochs):
>         model.train()
>         total, count = 0.0, 0
>         for xb, yb in loader:
>             xb, yb = xb.to(device), yb.to(device)
>             loss = F.mse_loss(model(xb), yb)
>             optimizer.zero_grad()
>             loss.backward()
>             optimizer.step()
>             total += loss.item() * len(xb)
>             count += len(xb)
>         history["loss"].append(total / count)
>         history["val_loss"].append(mse(Y_va, predict(model, X_va)))
>     n_params = sum(p.numel() for p in model.parameters())
>     print(f"{n_params} parameters, epoch {epochs}: "
>           f"train {history['loss'][-1]:.4f}, valid {history['val_loss'][-1]:.4f}")
>     return history
>
>
> set_seed(42)
> linear = nn.Sequential(nn.Flatten(), nn.Linear(n_steps, 1))
> history = fit(linear, X_train, y_train, X_valid, y_valid)
> results["Linear, 1 step"] = history["val_loss"][-1]
> ```

```output theme={null}
51 parameters, epoch 20: train 0.0037, valid 0.0036
```

![Output from cell 8](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_8_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=f57bd9314d88b9e398e810dbfaa2853b)

## The recurrent cell

The [simple RNN](https://aegean.ai/aiml-common/lectures/rnn/simple-rnn/index.md) updates its hidden state with

$$
\begin{aligned}
\mathbf a_t &= \mathbf W \mathbf h_{t-1} + \mathbf U \mathbf x_t + \mathbf b \\
\mathbf h_t &= \tanh(\mathbf a_t)
\end{aligned}
$$

and the same $\mathbf W$, $\mathbf U$ and $\mathbf b$ are reused at every step. The cell below implements these two lines as a module. A second module unrolls any cell over the time axis, starting from $\mathbf h_0 = \mathbf 0$, and returns the hidden state at every step.

```python theme={null}
class SimpleRNNCell(nn.Module):
    """h_t = tanh(W h_{t-1} + U x_t + b)"""

    def __init__(self, input_size, hidden_size):
        super().__init__()
        self.hidden_size = hidden_size
        self.U = nn.Linear(input_size, hidden_size)               # U x_t + b
        self.W = nn.Linear(hidden_size, hidden_size, bias=False)  # W h_{t-1}

    def forward(self, x_t, h_prev):
        a_t = self.W(h_prev) + self.U(x_t)
        return torch.tanh(a_t)


class Unroll(nn.Module):
    """Apply a cell along the time axis of a (batch, time, features) tensor."""

    def __init__(self, cell):
        super().__init__()
        self.cell = cell

    def forward(self, x):
        h = x.new_zeros(x.shape[0], self.cell.hidden_size)
        states = []
        for t in range(x.shape[1]):
            h = self.cell(x[:, t], h)
            states.append(h)
        return torch.stack(states, dim=1)
```

`nn.RNN` computes the same recurrence with a fused kernel. It stores $\mathbf U$ as `weight_ih_l0` and $\mathbf W$ as `weight_hh_l0`, and it splits $\mathbf b$ into two bias vectors that are added together. Copy those weights into the hand-written cell and the two produce the same hidden states, up to floating-point rounding.

> [accordion] Code
>
> ```python theme={null}
> set_seed(42)
> torch_rnn = nn.RNN(input_size=1, hidden_size=20, batch_first=True)
> scratch = Unroll(SimpleRNNCell(1, 20))
> with torch.no_grad():
>     scratch.cell.U.weight.copy_(torch_rnn.weight_ih_l0)
>     scratch.cell.U.bias.copy_(torch_rnn.bias_ih_l0 + torch_rnn.bias_hh_l0)
>     scratch.cell.W.weight.copy_(torch_rnn.weight_hh_l0)
>
> x = to_tensor(X_train[:5])
> h_torch, h_last = torch_rnn(x)
> h_scratch = scratch(x)
> print("hidden states:", tuple(h_scratch.shape))
> print(f"max |difference|: {(h_torch - h_scratch).abs().max().item():.1e}")
> print("last state of nn.RNN equals output at t = 50:", torch.equal(h_last[0], h_torch[:, -1]))
> ```

```output theme={null}
hidden states: (5, 50, 20)
max |difference|: 8.9e-08
last state of nn.RNN equals output at t = 50: True
```

## A simple RNN

The smallest possible RNN has a single recurrent neuron, one input and no output layer. Its hidden state at the last step is the forecast. It has three parameters, $w$, $u$ and $b$ (`nn.RNN` stores $b$ as two biases, so it reports four). Because $\tanh$ bounds the state to $(-1, 1)$, the forecast can never leave that range. This model is trained with a larger learning rate, 0.005.

> [accordion] Code
>
> ```python theme={null}
> class SequenceRegressor(nn.Module):
>     """A recurrent body followed by an optional linear head o = V h + c.
>
>     last_step_only=True feeds only h_tau to the head (sequence to vector);
>     False applies the head at every step (sequence to sequence).
>     """
>
>     def __init__(self, body, hidden_size=20, out_features=None, last_step_only=True):
>         super().__init__()
>         self.body = body
>         self.head = nn.Linear(hidden_size, out_features) if out_features else nn.Identity()
>         self.last_step_only = last_step_only
>
>     def forward(self, x):
>         h = self.body(x)
>         if isinstance(h, tuple):  # nn.RNN, nn.LSTM and nn.GRU return (outputs, final state)
>             h = h[0]
>         if self.last_step_only:
>             h = h[:, -1]
>         return self.head(h)
>
>
> set_seed(42)
> simple_rnn = SequenceRegressor(nn.RNN(1, 1, batch_first=True), hidden_size=1)
> history = fit(simple_rnn, X_train, y_train, X_valid, y_valid, lr=0.005)
> results["Simple RNN, 1 neuron"] = history["val_loss"][-1]
> ```

```output theme={null}
4 parameters, epoch 20: train 0.0114, valid 0.0109
```

![Output from cell 12](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_12_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=5cebf330dea67dd338231c4d5f46fb8c)

## Deep RNNs

The single-neuron RNN reaches an MSE of 0.0109. That is better than the naive forecast but about three times worse than the linear model (0.0036), which has 51 parameters to its 4.

Stacking recurrent layers gives a deep RNN. Each layer receives the full sequence of hidden states of the layer below. The first model stacks two layers of 20 neurons and a third recurrent layer with one neuron, whose last hidden state is the forecast.

> [accordion] Code
>
> ```python theme={null}
> class StackedRNN(nn.Module):
>     """Recurrent layers of possibly different widths, e.g. sizes=[1, 20, 20, 1]."""
>
>     def __init__(self, sizes):
>         super().__init__()
>         self.layers = nn.ModuleList(
>             nn.RNN(n_in, n_out, batch_first=True) for n_in, n_out in zip(sizes[:-1], sizes[1:])
>         )
>
>     def forward(self, x):
>         for layer in self.layers:
>             x, _ = layer(x)
>         return x
>
>
> set_seed(42)
> deep_rnn = SequenceRegressor(StackedRNN([1, 20, 20, 1]), hidden_size=1)
> history = fit(deep_rnn, X_train, y_train, X_valid, y_valid)
> results["Deep RNN, recurrent output"] = history["val_loss"][-1]
> ```

```output theme={null}
1323 parameters, epoch 20: train 0.0036, valid 0.0033
```

![Output from cell 14](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_14_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=2f3fe1857639c3df6a5a9fcde0e35a67)

The stacked RNN reaches 0.0033, slightly better than the linear model.

A single recurrent neuron makes a poor output layer. Its state must carry all the memory of the top layer, and it is squashed by $\tanh$. A better design keeps two recurrent layers of 20 neurons and adds a linear output layer, $o = \mathbf V \mathbf h_\tau + c$, on the last hidden state only. This is the sequence-to-vector arrangement.

> [accordion] Code
>
> ```python theme={null}
> set_seed(42)
> deep_rnn_head = SequenceRegressor(nn.RNN(1, 20, num_layers=2, batch_first=True),
>                                   hidden_size=20, out_features=1)
> history = fit(deep_rnn_head, X_train, y_train, X_valid, y_valid)
> results["Deep RNN, linear output"] = history["val_loss"][-1]
> ```

```output theme={null}
1321 parameters, epoch 20: train 0.0035, valid 0.0031
```

![Output from cell 16](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_16_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=f92dbb7a1524337f3efed65b3921563d)

## Forecasting several steps ahead

With the linear output layer the deep RNN reaches 0.0031, the lowest one-step error in this section.

To forecast ten steps, you can reuse the one-step model. Predict the next value, append it to the input, and predict again from the longer sequence. The RNN accepts sequences of any length, so it runs unchanged on the 51-, 52-, ... step inputs. Each prediction becomes part of the next input, so errors accumulate along the horizon.

> [accordion] Code
>
> ```python theme={null}
> def forecast_iteratively(model, X, ahead=10):
>     for _ in range(ahead):
>         y_next = predict(model, X)[:, np.newaxis, :]
>         X = np.concatenate([X, y_next], axis=1)
>     return X[:, -ahead:]
>
>
> np.random.seed(43)  # a new series, not one from the training set
> series = generate_time_series(1, n_steps + 10)
> X_new, Y_new = series[:, :n_steps], series[:, n_steps:]
> Y_pred = forecast_iteratively(deep_rnn_head, X_new)
> print("Y_pred:", Y_pred.shape)
> ```

```output theme={null}
Y_pred: (1, 10, 1)
```

![Output from cell 18](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_18_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=bfce1cc27ed3d6269498132b11e33c91)

To measure the error over the whole horizon, regenerate the dataset with ten target values per series, then forecast all 2,000 validation series one step at a time and compare with the two baselines. The naive baseline repeats the last observed value ten times. The linear baseline predicts all ten values at once with ten outputs.

> [accordion] Code
>
> ```python theme={null}
> np.random.seed(42)
> series = generate_time_series(10000, n_steps + 10)
> X_train, Y_train = series[:7000, :n_steps], series[:7000, -10:, 0]
> X_valid, Y_valid = series[7000:9000, :n_steps], series[7000:9000, -10:, 0]
> X_test, Y_test = series[9000:, :n_steps], series[9000:, -10:, 0]
>
> Y_pred = forecast_iteratively(deep_rnn_head, X_valid)[..., 0]
> results["Deep RNN, iterated 10 times"] = mse(Y_valid, Y_pred)
> results["Naive, 10 steps"] = mse(Y_valid, np.tile(X_valid[:, -1], 10))
> print(f"Iterated one-step RNN MSE: {results['Deep RNN, iterated 10 times']:.4f}")
> print(f"Naive forecast MSE:        {results['Naive, 10 steps']:.4f}")
>
> set_seed(42)
> linear10 = nn.Sequential(nn.Flatten(), nn.Linear(n_steps, 10))
> history = fit(linear10, X_train, Y_train, X_valid, Y_valid)
> results["Linear, 10 steps"] = history["val_loss"][-1]
> ```

```output theme={null}
Iterated one-step RNN MSE: 0.0368
Naive forecast MSE:        0.2570
510 parameters, epoch 20: train 0.0179, valid 0.0179
```

Over the ten-step horizon, the iterated RNN has an MSE of 0.0368. That is far better than the naive forecast (0.2570) but worse than the linear model (0.0179), because the errors of the early predictions compound.

An RNN can also predict all ten values in one pass. The deep RNN keeps its structure and its linear output layer gets ten outputs instead of one, $\hat{\mathbf y} = \mathbf V \mathbf h_\tau + \mathbf c$ with $\hat{\mathbf y} \in \mathbb R^{10}$.

> [accordion] Code
>
> ```python theme={null}
> set_seed(42)
> rnn10 = SequenceRegressor(nn.RNN(1, 20, num_layers=2, batch_first=True),
>                           hidden_size=20, out_features=10)
> history = fit(rnn10, X_train, Y_train, X_valid, Y_valid)
> results["Deep RNN, 10 outputs"] = history["val_loss"][-1]
>
> np.random.seed(43)
> series = generate_time_series(1, n_steps + 10)
> X_new, Y_new = series[:, :n_steps], series[:, n_steps:]
> Y_pred = predict(rnn10, X_new)[..., np.newaxis]
> ```

```output theme={null}
1510 parameters, epoch 20: train 0.0111, valid 0.0099
```

![Output from cell 21](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_21_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=33c7a0a99b5b3f63712b58a1c492435f)

## Sequence-to-sequence forecasting

The RNN with ten outputs reaches 0.0099, about half the error of the linear model.

The model above computes a loss only at the last step. You can give it a training signal at every step instead. At each step $t$ the model forecasts the next ten values $x(t+1), \dots, x(t+10)$, so at $t = 0$ it forecasts steps 1 to 10, at $t = 1$ steps 2 to 11, and at the last step, steps 50 to 59. The model stays causal: at step $t$ its hidden state has seen only $x(0), \dots, x(t)$.

The targets become a tensor of shape (series, 50, 10), and the linear head is applied to the hidden state at every step. Gradients now flow into the network from all 50 steps instead of only the last one. Only the forecast made at the last step matters at prediction time, so the models below also report the error of that last step.

> [accordion] Code
>
> ```python theme={null}
> np.random.seed(42)
> series = generate_time_series(10000, n_steps + 10)
> X_train, X_valid, X_test = series[:7000, :n_steps], series[7000:9000, :n_steps], series[9000:, :n_steps]
> Y = np.empty((10000, n_steps, 10), dtype=np.float32)
> for step_ahead in range(1, 10 + 1):
>     Y[..., step_ahead - 1] = series[..., step_ahead:step_ahead + n_steps, 0]
> Y_train, Y_valid, Y_test = Y[:7000], Y[7000:9000], Y[9000:]
> print("X_train:", X_train.shape, " Y_train:", Y_train.shape)
>
> np.random.seed(43)
> series = generate_time_series(1, n_steps + 10)
> X_new, Y_new = series[:, :n_steps], series[:, n_steps:]
>
>
> def last_time_step_mse(model):
>     return mse(Y_valid[:, -1], predict(model, X_valid)[:, -1])
>
>
> def train_seq2seq(name, body, lr=1e-3):
>     set_seed(42)
>     model = SequenceRegressor(body, hidden_size=20, out_features=10, last_step_only=False)
>     history = fit(model, X_train, Y_train, X_valid, Y_valid, lr=lr)
>     results[name] = last_time_step_mse(model)
>     print(f"last-time-step MSE: {results[name]:.4f}")
>     return model, history
>
>
> seq2seq, history = train_seq2seq("Deep RNN, seq2seq",
>                                  nn.RNN(1, 20, num_layers=2, batch_first=True), lr=0.01)
> ```

```output theme={null}
X_train: (7000, 50, 1)  Y_train: (7000, 50, 10)
1510 parameters, epoch 20: train 0.0182, valid 0.0173
last-time-step MSE: 0.0061
```

![Output from cell 23](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_23_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=3865370b756eccc125ff15c84bbcd2b3)

## Normalization in deep RNNs

The sequence-to-sequence RNN brings the error of the last-step forecast down to 0.0061. It trains with a learning rate of 0.01. The models below use the default 0.001.

Batch normalization standardizes each feature with statistics computed over the mini-batch. In an RNN those statistics are pooled over the batch and over all time steps, and they can only be inserted between recurrent layers, not inside the recurrence. The next model places a batch-norm layer after each of the two recurrent layers.

> [accordion] Code
>
> ```python theme={null}
> class BatchNormDeepRNN(nn.Module):
>     def __init__(self, hidden_size=20):
>         super().__init__()
>         self.rnn1 = nn.RNN(1, hidden_size, batch_first=True)
>         self.bn1 = nn.BatchNorm1d(hidden_size)
>         self.rnn2 = nn.RNN(hidden_size, hidden_size, batch_first=True)
>         self.bn2 = nn.BatchNorm1d(hidden_size)
>
>     def forward(self, x):
>         h, _ = self.rnn1(x)
>         h = self.bn1(h.transpose(1, 2)).transpose(1, 2)  # BatchNorm1d wants (batch, features, time)
>         h, _ = self.rnn2(h)
>         return self.bn2(h.transpose(1, 2)).transpose(1, 2)
>
>
> bn_rnn, history = train_seq2seq("Deep RNN, batch norm", BatchNormDeepRNN())
> ```

```output theme={null}
1590 parameters, epoch 20: train 0.0206, valid 0.0281
last-time-step MSE: 0.0181
```

With batch normalization the last-step error is 0.0181, so it does not help on this task.

*Layer normalization* fits recurrences better. It standardizes each example across its features, so it needs no batch statistics and behaves the same at every time step and at training and inference time. It can sit inside the cell, between the affine map and the non-linearity:

$$
\begin{aligned}
\mathbf h_t &= \tanh\big(\mathrm{LN}(\mathbf W \mathbf h_{t-1} + \mathbf U \mathbf x_t + \mathbf b)\big) \\
\mathrm{LN}(\mathbf a) &= \boldsymbol\gamma \odot \frac{\mathbf a - \mu}{\sigma} + \boldsymbol\beta
\end{aligned}
$$

where $\mu$ and $\sigma$ are the mean and standard deviation of the entries of $\mathbf a$, and $\boldsymbol\gamma$, $\boldsymbol\beta$ are learned. `nn.RNN` has no hook for this, so the model uses the hand-written cell from earlier with one extra line, unrolled by the same `Unroll` module.

```python theme={null}
class LayerNormRNNCell(SimpleRNNCell):
    """h_t = tanh(LN(W h_{t-1} + U x_t + b))"""

    def __init__(self, input_size, hidden_size):
        super().__init__(input_size, hidden_size)
        self.layer_norm = nn.LayerNorm(hidden_size)

    def forward(self, x_t, h_prev):
        a_t = self.W(h_prev) + self.U(x_t)
        return torch.tanh(self.layer_norm(a_t))
```

> [accordion] Code
>
> ```python theme={null}
> ln_rnn, history = train_seq2seq(
>     "Deep RNN, layer norm cell",
>     nn.Sequential(Unroll(LayerNormRNNCell(1, 20)), Unroll(LayerNormRNNCell(20, 20))),
> )
> ```

```output theme={null}
1550 parameters, epoch 20: train 0.0230, valid 0.0227
last-time-step MSE: 0.0113
```

## LSTM

The layer-norm cell reaches 0.0113, better than batch normalization at the same learning rate.

The [LSTM](https://aegean.ai/aiml-common/lectures/rnn/lstm/index.md) replaces the simple cell with a gated one that keeps a separate cell state. `nn.LSTM` takes the same arguments as `nn.RNN`, so the sequence-to-sequence model only changes its body. With 20 units, an LSTM layer has four times the weights of a simple RNN layer, one set for each gate and one for the candidate update.

> [accordion] Code
>
> ```python theme={null}
> lstm, history = train_seq2seq("LSTM, seq2seq", nn.LSTM(1, 20, num_layers=2, batch_first=True))
> ```

```output theme={null}
5410 parameters, epoch 20: train 0.0198, valid 0.0197
last-time-step MSE: 0.0057
```

![Output from cell 28](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_28_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=b8d44127ab4d034e52c16abc73fa8d76)

## GRU

The LSTM reaches a last-step error of 0.0057, the lowest ten-step error in this section.

The gated recurrent unit (GRU) simplifies the LSTM. It merges the cell state into the hidden state and uses two gates instead of three. An *update gate* $\mathbf z_t$ decides how much of the old state to keep, and a *reset gate* $\mathbf r_t$ decides how much of the old state enters the candidate state $\tilde{\mathbf h}_t$:

$$
\begin{aligned}
\mathbf z_t &= \sigma(\mathbf W_z \mathbf h_{t-1} + \mathbf U_z \mathbf x_t + \mathbf b_z) \\
\mathbf r_t &= \sigma(\mathbf W_r \mathbf h_{t-1} + \mathbf U_r \mathbf x_t + \mathbf b_r) \\
\tilde{\mathbf h}_t &= \tanh\big(\mathbf W (\mathbf r_t \odot \mathbf h_{t-1}) + \mathbf U \mathbf x_t + \mathbf b\big) \\
\mathbf h_t &= \mathbf z_t \odot \mathbf h_{t-1} + (1 - \mathbf z_t) \odot \tilde{\mathbf h}_t
\end{aligned}
$$

The last line plays the role of the LSTM cell-state update: the state is a gated mix of its old value and a new candidate. `nn.GRU` places the reset gate after the multiplication by $\mathbf W$, a small variant of the same idea.

> [accordion] Code
>
> ```python theme={null}
> gru, history = train_seq2seq("GRU, seq2seq", nn.GRU(1, 20, num_layers=2, batch_first=True))
> ```

```output theme={null}
4110 parameters, epoch 20: train 0.0223, valid 0.0221
last-time-step MSE: 0.0082
```

![Output from cell 30](https://mintcdn.com/aegeanaiinc/3EuecqmCeDvLuk48/aiml-common/lectures/rnn/time-series-forecasting/images/cell_30_output_1.svg?fit=max\&auto=format\&n=3EuecqmCeDvLuk48\&q=85\&s=5518274386836f951fe36d3636afef59)

## Comparison

The GRU reaches 0.0082 with 4,110 parameters, against 5,410 for the LSTM.

The table collects the validation errors from this section. The one-step and ten-step rows measure different tasks and cannot be compared with each other. For the sequence-to-sequence models the error is that of the forecast made at the last step, which is the same task as the other ten-step rows. The learning rate differs between rows (0.005 for the single neuron, 0.01 for the plain sequence-to-sequence RNN, 0.001 elsewhere) and every model ran for only 20 epochs, so small gaps between the gated and normalized models are not a ranking of the architectures.

> [accordion] Code
>
> ```python theme={null}
> print(f"{'Model':<30}{'Validation MSE':>15}")
> print("-" * 45)
> for name, value in results.items():
>     print(f"{name:<30}{value:>15.4f}")
> ```

```output theme={null}
Model                          Validation MSE
---------------------------------------------
Naive, 1 step                          0.0202
Linear, 1 step                         0.0036
Simple RNN, 1 neuron                   0.0109
Deep RNN, recurrent output             0.0033
Deep RNN, linear output                0.0031
Deep RNN, iterated 10 times            0.0368
Naive, 10 steps                        0.2570
Linear, 10 steps                       0.0179
Deep RNN, 10 outputs                   0.0099
Deep RNN, seq2seq                      0.0061
Deep RNN, batch norm                   0.0181
Deep RNN, layer norm cell              0.0113
LSTM, seq2seq                          0.0057
GRU, seq2seq                           0.0082
```

## PyTorch reference

| PyTorch class | Description |
| - | - |
| [`nn.RNN`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.RNN.html) | Multi-layer Elman RNN with tanh or ReLU. |
| [`nn.LSTM`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.LSTM.html) | Multi-layer long short-term memory RNN. |
| [`nn.GRU`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.GRU.html) | Multi-layer gated recurrent unit RNN. |
| [`nn.LayerNorm`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.LayerNorm.html) | Layer normalization over the last dimensions. |
| [`nn.BatchNorm1d`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.BatchNorm1d.html) | Batch normalization over 2D or 3D input. |
| [`nn.Linear`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Linear.html) | Affine transformation $y = xA^T + b$. |

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/lectures/rnn/time-series-forecasting).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.