Skip to main content
The simple RNN suffers from exploding or vanishing gradients when the span of backpropagation is large (large τ\tau). The IIR filter analogy showed that a recurrence integrates its input, and that you have to control the weight ww to keep the impulse response from exploding or vanishing. Gated RNNs do exactly this. The best known gated RNN is the long short-term memory (LSTM) network. In an LSTM the weight ww is not fixed: the network computes it from the context of the input sequence. The architecture is shown below. LSTM cell with input, forget and output gates around an inner cell-state loop LSTM architecture, divided into three areas: input (green), cell state (blue) and output (red). You can see the outer recurrence loop through ht−1\mathbf h_{t-1} and the inner loop through st−1\mathbf s_{t-1}. The multipliers are Hadamard (element-wise) products. If you set the input and output gates to 1 and the forget gate to 0, you are back to a simple RNN layer. To capture an input context that reaches several time steps into the past, the LSTM adds an inner recurrence loop. This loop acts as a variable-length memory inside the cell, called the cell state. A gate called the forget gate learns the context and sets the forgetting factor, which plays the role of ww in the IIR filter. It decides how much of the previous cell state the cell keeps. Two more gates, the input gate and the output gate, complete the cell. The components below appear in the order in which the cell processes them.
  1. The index tt runs over the unrolled sequence and the index ii runs over the cells of the LSTM hidden layer.
  2. A(i,:)\mathbf A(i,:) denotes the ii-th row of the matrix A\mathbf A, and bib^i denotes the ii-th element of the vector b\mathbf b.

Forward pass

Input gate

The input gate protects the contents of the cell state from inputs that are irrelevant to the context. The LSTM learns when to let activations from the input layer into the internal state. While the input gate is zero, no activation gets in. The input gate computes the factor gti=σ(Wg(i,:) ht−1+Ug(i,:) xt+bgi)g_t^i = \sigma \big( \mathbf W_g(i,:)\, \mathbf h_{t-1} + \mathbf U_g(i,:)\, \mathbf x_t + b_g^i \big) The sigmoid produces a factor between 0 and 1 for each cell ii of the LSTM hidden layer. The factor scales the input of cell ii. That input combines the previous hidden state, through the dot product Wg(i,:) ht−1\mathbf W_g(i,:)\, \mathbf h_{t-1}, and the current input, through the dot product Ug(i,:) xt\mathbf U_g(i,:)\, \mathbf x_t.

Input layer

The input layer is formed exactly like the input of the simple RNN layer in the previous section.

Forget gate

The forget gate computes the forgetting factor fti=σ(Wf(i,:) ht−1+Uf(i,:) xt+bfi)f_t^i = \sigma \big( \mathbf W_f(i,:)\, \mathbf h_{t-1} + \mathbf U_f(i,:)\, \mathbf x_t + b_f^i \big) Like the input gate factor, it lies between 0 and 1. It sets how much of the earlier cell state is preserved.

Output gate

The output gate protects the subsequent cells from a hidden state that is irrelevant to their context. qti=σ(Wo(i,:) ht−1+Uo(i,:) xt+boi)q_t^i = \sigma \big( \mathbf W_o(i,:)\, \mathbf h_{t-1} + \mathbf U_o(i,:)\, \mathbf x_t + b_o^i \big) The output gate learns when to let the hidden state out.

Cell state

The cell state is the core of the LSTM cell. It is the new memory that the LSTM introduces, and the gate factors above exist to preserve it for as long as the task needs it. The update is ati=W(i,:) ht−1+U(i,:) xt+bisti=fti st−1i+gtitanh⁡(ati)\begin{aligned} a_t^i &= \mathbf W(i,:)\, \mathbf h_{t-1} + \mathbf U(i,:)\, \mathbf x_t + b^i \\ s_t^i &= f_t^i\, s_{t-1}^i + g_t^i \tanh ( a_t^i ) \end{aligned} The parameters θin={W,U,b}\boldsymbol\theta_{in} = \{ \mathbf W, \mathbf U, \mathbf b \} are the recurrent weights, the input weights and the bias at the input of the cells.

Hidden state

The hidden state is the output of the LSTM cell. It depends on the cell state and the output gate: hti=qtitanh⁡(sti)h_t^i = q_t^i \tanh(s_t^i) If the input and output gates are both closed (equal to 0) and the forget gate is fully open (equal to 1), the activation is trapped in the memory cell. It neither grows nor shrinks, and it does not affect the output at intermediate time steps.

Output layer

The output layer is the same as in the simple RNN. Depending on the task (classification, regression and so on), it is typically a fully connected layer followed by a softmax or another output unit.

Backward pass

Backpropagation works as in the simple RNN, with more complicated expressions. The additive update of the cell state forms what is called the constant error carousel. It lets the gradient flow back across many time steps without exploding or vanishing. In this sense the gates learn when to let the error in and when to let it out. In practice, LSTMs learn long-range dependencies much better than simple RNNs. The time series section that follows trains an LSTM and compares it with simple RNNs on the same data. Greff et al. study the LSTM design and its hyperparameters systematically in “LSTM: A Search Space Odyssey”.

Additional resources

These tutorials cover LSTMs from other angles:
  1. A Critical Review of Recurrent Neural Networks for Sequence Learning
  2. Understanding LSTMs
  3. Illustrated guide to LSTMs
  4. Simplest possible LSTM explanation video

PyTorch reference

Key references: (Sak et al., 2014; Collins et al., 2016; Sutskever et al., 2014; Zaremba et al., 2014; Jaderberg et al., 2016)

References

  • Collins, J., Sohl-Dickstein, J., Sussillo, D. (2016). Capacity and Trainability in Recurrent Neural Networks. arXiv [stat.ML]. Link
  • Jaderberg, M., Czarnecki, W., Osindero, S., Vinyals, O., Graves, A., et al. (2016). Decoupled Neural Interfaces using Synthetic Gradients. arXiv [cs.LG]. Link
  • Sak, H., Senior, A., Beaufays, F. (2014). Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition. arXiv [cs.NE]. Link
  • Sutskever, I., Vinyals, O., Le, Q. (2014). Sequence to Sequence Learning with Neural Networks. arXiv [cs.CL]. Link
  • Zaremba, W., Sutskever, I., Vinyals, O. (2014). Recurrent Neural Network Regularization. arXiv [cs.NE]. Link

Found an error on this page? Report an issue.