Code
Code
The dataset
Each example is a sum of two sine waves with random frequencies and phases, plus a little uniform noise. All series have the same length and live roughly in : where are drawn uniformly from once per series, and is drawn uniformly from at every step. Each series is sampled at equally spaced times in . The input is the first 50 values of a series and the target is value 51. The 10,000 series are split into 7,000 for training, 2,000 for validation and 1,000 for testing. The input tensor has shape (batch, time steps, features), here with a single feature per step.Code
Code
Baselines
A forecaster is only useful if it beats simple rules. The naive forecast predicts that the next value equals the last observed value, . All errors in this section are mean squared errors (MSE) on the validation set.Code
Code
Code
Code
The recurrent cell
The simple RNN updates its hidden state with and the same , and are reused at every step. The cell below implements these two lines as a module. A second module unrolls any cell over the time axis, starting from , and returns the hidden state at every step.nn.RNN computes the same recurrence with a fused kernel. It stores as weight_ih_l0 and as weight_hh_l0, and it splits into two bias vectors that are added together. Copy those weights into the hand-written cell and the two produce the same hidden states, up to floating-point rounding.
Code
Code
A simple RNN
The smallest possible RNN has a single recurrent neuron, one input and no output layer. Its hidden state at the last step is the forecast. It has three parameters, , and (nn.RNN stores as two biases, so it reports four). Because bounds the state to , the forecast can never leave that range. This model is trained with a larger learning rate, 0.005.
Code
Code
Deep RNNs
The single-neuron RNN reaches an MSE of 0.0109. That is better than the naive forecast but about three times worse than the linear model (0.0036), which has 51 parameters to its 4. Stacking recurrent layers gives a deep RNN. Each layer receives the full sequence of hidden states of the layer below. The first model stacks two layers of 20 neurons and a third recurrent layer with one neuron, whose last hidden state is the forecast.Code
Code
Code
Code
Forecasting several steps ahead
With the linear output layer the deep RNN reaches 0.0031, the lowest one-step error in this section. To forecast ten steps, you can reuse the one-step model. Predict the next value, append it to the input, and predict again from the longer sequence. The RNN accepts sequences of any length, so it runs unchanged on the 51-, 52-, … step inputs. Each prediction becomes part of the next input, so errors accumulate along the horizon.Code
Code
Code
Code
Code
Code
Sequence-to-sequence forecasting
The RNN with ten outputs reaches 0.0099, about half the error of the linear model. The model above computes a loss only at the last step. You can give it a training signal at every step instead. At each step the model forecasts the next ten values , so at it forecasts steps 1 to 10, at steps 2 to 11, and at the last step, steps 50 to 59. The model stays causal: at step its hidden state has seen only . The targets become a tensor of shape (series, 50, 10), and the linear head is applied to the hidden state at every step. Gradients now flow into the network from all 50 steps instead of only the last one. Only the forecast made at the last step matters at prediction time, so the models below also report the error of that last step.Code
Code
Normalization in deep RNNs
The sequence-to-sequence RNN brings the error of the last-step forecast down to 0.0061. It trains with a learning rate of 0.01. The models below use the default 0.001. Batch normalization standardizes each feature with statistics computed over the mini-batch. In an RNN those statistics are pooled over the batch and over all time steps, and they can only be inserted between recurrent layers, not inside the recurrence. The next model places a batch-norm layer after each of the two recurrent layers.Code
Code
nn.RNN has no hook for this, so the model uses the hand-written cell from earlier with one extra line, unrolled by the same Unroll module.
Code
Code
LSTM
The layer-norm cell reaches 0.0113, better than batch normalization at the same learning rate. The LSTM replaces the simple cell with a gated one that keeps a separate cell state.nn.LSTM takes the same arguments as nn.RNN, so the sequence-to-sequence model only changes its body. With 20 units, an LSTM layer has four times the weights of a simple RNN layer, one set for each gate and one for the candidate update.
Code
Code
GRU
The LSTM reaches a last-step error of 0.0057, the lowest ten-step error in this section. The gated recurrent unit (GRU) simplifies the LSTM. It merges the cell state into the hidden state and uses two gates instead of three. An update gate decides how much of the old state to keep, and a reset gate decides how much of the old state enters the candidate state : The last line plays the role of the LSTM cell-state update: the state is a gated mix of its old value and a new candidate.nn.GRU places the reset gate after the multiplication by , a small variant of the same idea.
Code
Code
Comparison
The GRU reaches 0.0082 with 4,110 parameters, against 5,410 for the LSTM. The table collects the validation errors from this section. The one-step and ten-step rows measure different tasks and cannot be compared with each other. For the sequence-to-sequence models the error is that of the forecast made at the last step, which is the same task as the other ten-step rows. The learning rate differs between rows (0.005 for the single neuron, 0.01 for the plain sequence-to-sequence RNN, 0.001 elsewhere) and every model ran for only 20 epochs, so small gaps between the gated and normalized models are not a ranking of the architectures.Code
Code
PyTorch reference
Found an error on this page? Report an issue.

