Skip to main content
Open In Colab In this section you train recurrent networks to forecast a time series. You start from two baselines that any useful model must beat, build the cell of the simple RNN section by hand, and then train deeper RNNs, an LSTM and a GRU on the same data. The models predict one step ahead first, and then ten steps ahead. The examples follow chapter 15 of Aurélien Géron’s Hands-On Machine Learning with Scikit-Learn, Keras and TensorFlow (2nd edition), rewritten for PyTorch.

The dataset

Each example is a sum of two sine waves with random frequencies and phases, plus a little uniform noise. All series have the same length and live roughly in [−1,1][-1, 1]: x(t)=0.5sin⁡((t−o1)(10+10f1))+0.2sin⁡((t−o2)(20+20f2))+ϵt\begin{aligned} x(t) = {} & 0.5 \sin\big((t - o_1)(10 + 10 f_1)\big) \\ & + 0.2 \sin\big((t - o_2)(20 + 20 f_2)\big) + \epsilon_t \end{aligned} where f1,f2,o1,o2f_1, f_2, o_1, o_2 are drawn uniformly from [0,1)[0, 1) once per series, and ϵt\epsilon_t is drawn uniformly from [−0.05,0.05)[-0.05, 0.05) at every step. Each series is sampled at equally spaced times tt in [0,1][0, 1]. The input is the first 50 values of a series and the target is value 51. The 10,000 series are split into 7,000 for training, 2,000 for validation and 1,000 for testing. The input tensor has shape (batch, time steps, features), here with a single feature per step.
Three validation series and the value that follows each of them. The gray dots are the 50 inputs and the blue dot is the target. In the figures that follow, blue marks targets and orange crosses mark predictions. Output from cell 4

Baselines

A forecaster is only useful if it beats simple rules. The naive forecast predicts that the next value equals the last observed value, x^(51)=x(50)\hat x(51) = x(50). All errors in this section are mean squared errors (MSE) on the validation set.
Output from cell 6 The naive forecast has a validation MSE of 0.0202. The second baseline is a linear model. It flattens the 50 inputs into a vector and predicts the next value as a weighted sum, x^(51)=w⊤x+b\hat x(51) = \mathbf w^\top \mathbf x + b, with 51 parameters. Every model in this section trains the same way: Adam on the MSE loss, mini-batches of 32 series, 20 epochs.
Output from cell 8

The recurrent cell

The simple RNN updates its hidden state with at=Wht−1+Uxt+bht=tanh⁡(at)\begin{aligned} \mathbf a_t &= \mathbf W \mathbf h_{t-1} + \mathbf U \mathbf x_t + \mathbf b \\ \mathbf h_t &= \tanh(\mathbf a_t) \end{aligned} and the same W\mathbf W, U\mathbf U and b\mathbf b are reused at every step. The cell below implements these two lines as a module. A second module unrolls any cell over the time axis, starting from h0=0\mathbf h_0 = \mathbf 0, and returns the hidden state at every step.
nn.RNN computes the same recurrence with a fused kernel. It stores U\mathbf U as weight_ih_l0 and W\mathbf W as weight_hh_l0, and it splits b\mathbf b into two bias vectors that are added together. Copy those weights into the hand-written cell and the two produce the same hidden states, up to floating-point rounding.

A simple RNN

The smallest possible RNN has a single recurrent neuron, one input and no output layer. Its hidden state at the last step is the forecast. It has three parameters, ww, uu and bb (nn.RNN stores bb as two biases, so it reports four). Because tanh⁡\tanh bounds the state to (−1,1)(-1, 1), the forecast can never leave that range. This model is trained with a larger learning rate, 0.005.
Output from cell 12

Deep RNNs

The single-neuron RNN reaches an MSE of 0.0109. That is better than the naive forecast but about three times worse than the linear model (0.0036), which has 51 parameters to its 4. Stacking recurrent layers gives a deep RNN. Each layer receives the full sequence of hidden states of the layer below. The first model stacks two layers of 20 neurons and a third recurrent layer with one neuron, whose last hidden state is the forecast.
Output from cell 14 The stacked RNN reaches 0.0033, slightly better than the linear model. A single recurrent neuron makes a poor output layer. Its state must carry all the memory of the top layer, and it is squashed by tanh⁡\tanh. A better design keeps two recurrent layers of 20 neurons and adds a linear output layer, o=Vhτ+co = \mathbf V \mathbf h_\tau + c, on the last hidden state only. This is the sequence-to-vector arrangement.
Output from cell 16

Forecasting several steps ahead

With the linear output layer the deep RNN reaches 0.0031, the lowest one-step error in this section. To forecast ten steps, you can reuse the one-step model. Predict the next value, append it to the input, and predict again from the longer sequence. The RNN accepts sequences of any length, so it runs unchanged on the 51-, 52-, … step inputs. Each prediction becomes part of the next input, so errors accumulate along the horizon.
Output from cell 18 To measure the error over the whole horizon, regenerate the dataset with ten target values per series, then forecast all 2,000 validation series one step at a time and compare with the two baselines. The naive baseline repeats the last observed value ten times. The linear baseline predicts all ten values at once with ten outputs.
Over the ten-step horizon, the iterated RNN has an MSE of 0.0368. That is far better than the naive forecast (0.2570) but worse than the linear model (0.0179), because the errors of the early predictions compound. An RNN can also predict all ten values in one pass. The deep RNN keeps its structure and its linear output layer gets ten outputs instead of one, y^=Vhτ+c\hat{\mathbf y} = \mathbf V \mathbf h_\tau + \mathbf c with y^∈R10\hat{\mathbf y} \in \mathbb R^{10}.
Output from cell 21

Sequence-to-sequence forecasting

The RNN with ten outputs reaches 0.0099, about half the error of the linear model. The model above computes a loss only at the last step. You can give it a training signal at every step instead. At each step tt the model forecasts the next ten values x(t+1),…,x(t+10)x(t+1), \dots, x(t+10), so at t=0t = 0 it forecasts steps 1 to 10, at t=1t = 1 steps 2 to 11, and at the last step, steps 50 to 59. The model stays causal: at step tt its hidden state has seen only x(0),…,x(t)x(0), \dots, x(t). The targets become a tensor of shape (series, 50, 10), and the linear head is applied to the hidden state at every step. Gradients now flow into the network from all 50 steps instead of only the last one. Only the forecast made at the last step matters at prediction time, so the models below also report the error of that last step.
Output from cell 23

Normalization in deep RNNs

The sequence-to-sequence RNN brings the error of the last-step forecast down to 0.0061. It trains with a learning rate of 0.01. The models below use the default 0.001. Batch normalization standardizes each feature with statistics computed over the mini-batch. In an RNN those statistics are pooled over the batch and over all time steps, and they can only be inserted between recurrent layers, not inside the recurrence. The next model places a batch-norm layer after each of the two recurrent layers.
With batch normalization the last-step error is 0.0181, so it does not help on this task. Layer normalization fits recurrences better. It standardizes each example across its features, so it needs no batch statistics and behaves the same at every time step and at training and inference time. It can sit inside the cell, between the affine map and the non-linearity: ht=tanh⁡(LN(Wht−1+Uxt+b))LN(a)=γ⊙a−μσ+β\begin{aligned} \mathbf h_t &= \tanh\big(\mathrm{LN}(\mathbf W \mathbf h_{t-1} + \mathbf U \mathbf x_t + \mathbf b)\big) \\ \mathrm{LN}(\mathbf a) &= \boldsymbol\gamma \odot \frac{\mathbf a - \mu}{\sigma} + \boldsymbol\beta \end{aligned} where μ\mu and σ\sigma are the mean and standard deviation of the entries of a\mathbf a, and γ\boldsymbol\gamma, β\boldsymbol\beta are learned. nn.RNN has no hook for this, so the model uses the hand-written cell from earlier with one extra line, unrolled by the same Unroll module.

LSTM

The layer-norm cell reaches 0.0113, better than batch normalization at the same learning rate. The LSTM replaces the simple cell with a gated one that keeps a separate cell state. nn.LSTM takes the same arguments as nn.RNN, so the sequence-to-sequence model only changes its body. With 20 units, an LSTM layer has four times the weights of a simple RNN layer, one set for each gate and one for the candidate update.
Output from cell 28

GRU

The LSTM reaches a last-step error of 0.0057, the lowest ten-step error in this section. The gated recurrent unit (GRU) simplifies the LSTM. It merges the cell state into the hidden state and uses two gates instead of three. An update gate zt\mathbf z_t decides how much of the old state to keep, and a reset gate rt\mathbf r_t decides how much of the old state enters the candidate state h~t\tilde{\mathbf h}_t: zt=σ(Wzht−1+Uzxt+bz)rt=σ(Wrht−1+Urxt+br)h~t=tanh⁡(W(rt⊙ht−1)+Uxt+b)ht=zt⊙ht−1+(1−zt)⊙h~t\begin{aligned} \mathbf z_t &= \sigma(\mathbf W_z \mathbf h_{t-1} + \mathbf U_z \mathbf x_t + \mathbf b_z) \\ \mathbf r_t &= \sigma(\mathbf W_r \mathbf h_{t-1} + \mathbf U_r \mathbf x_t + \mathbf b_r) \\ \tilde{\mathbf h}_t &= \tanh\big(\mathbf W (\mathbf r_t \odot \mathbf h_{t-1}) + \mathbf U \mathbf x_t + \mathbf b\big) \\ \mathbf h_t &= \mathbf z_t \odot \mathbf h_{t-1} + (1 - \mathbf z_t) \odot \tilde{\mathbf h}_t \end{aligned} The last line plays the role of the LSTM cell-state update: the state is a gated mix of its old value and a new candidate. nn.GRU places the reset gate after the multiplication by W\mathbf W, a small variant of the same idea.
Output from cell 30

Comparison

The GRU reaches 0.0082 with 4,110 parameters, against 5,410 for the LSTM. The table collects the validation errors from this section. The one-step and ten-step rows measure different tasks and cannot be compared with each other. For the sequence-to-sequence models the error is that of the forecast made at the last step, which is the same task as the other ten-step rows. The learning rate differs between rows (0.005 for the single neuron, 0.01 for the plain sequence-to-sequence RNN, 0.001 elsewhere) and every model ran for only 20 epochs, so small gaps between the gated and normalized models are not a ranking of the architectures.

PyTorch reference


Found an error on this page? Report an issue.