> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Long Short-Term Memory (LSTM)

> The LSTM cell and its gates.

The simple RNN suffers from exploding or vanishing gradients when [the span of backpropagation is large](https://www.researchgate.net/publication/5583935_Learning_long-term_dependencies_with_gradient_descent_is_difficult) (large $\tau$). The IIR filter analogy showed that a recurrence *integrates* its input, and that you have to control the weight $w$ to keep the impulse response from exploding or vanishing.

Gated RNNs do exactly this. The best known gated RNN is the long short-term memory (LSTM) network. In an LSTM the weight $w$ is not fixed: the network computes it from the *context* of the input sequence. The architecture is shown below.

![LSTM cell with input, forget and output gates around an inner cell-state loop](https://mintcdn.com/aegeanaiinc/kzxVP5xXTKVusMDe/aiml-common/lectures/rnn/lstm/images/lstm.png?fit=max\&auto=format\&n=kzxVP5xXTKVusMDe\&q=85\&s=49aaffc8ab3de5a692410fce4eb7265a)

*LSTM architecture, divided into three areas: input (green), cell state (blue) and output (red). You can see the outer recurrence loop through $\mathbf h_{t-1}$ and the inner loop through $\mathbf s_{t-1}$. The multipliers are Hadamard (element-wise) products. If you set the input and output gates to 1 and the forget gate to 0, you are back to a simple RNN layer.*

To capture an input context that reaches several time steps into the past, the LSTM adds an inner recurrence loop. This loop acts as a variable-length memory inside the cell, called the *cell state*. A gate called the *forget gate* learns the context and sets the forgetting factor, which plays the role of $w$ in the IIR filter. It decides how much of the previous cell state the cell keeps. Two more gates, the *input gate* and the *output gate*, complete the cell.

The components below appear in the order in which the cell processes them.

> [!NOTE]
>
> 1. The index $t$ runs over the unrolled sequence and the index $i$ runs over the cells of the LSTM hidden layer.
>
> 2. $\mathbf A(i,:)$ denotes the $i$-th row of the matrix $\mathbf A$, and $b^i$ denotes the $i$-th element of the vector $\mathbf b$.

## Forward pass

### Input gate

The input gate *protects the contents of the cell state* from inputs that are irrelevant to the context. The LSTM learns when to let activations from the input layer into the internal state. While the input gate is zero, no activation gets in.

The input gate computes the factor

$$
g_t^i = \sigma \big( \mathbf W_g(i,:)\, \mathbf h_{t-1} + \mathbf U_g(i,:)\, \mathbf x_t + b_g^i \big)
$$

The sigmoid produces a factor between 0 and 1 for *each* cell $i$ of the LSTM hidden layer. The factor scales the input of cell $i$. That input combines the previous hidden state, through the dot product $\mathbf W_g(i,:)\, \mathbf h_{t-1}$, and the current input, through the dot product $\mathbf U_g(i,:)\, \mathbf x_t$.

### Input layer

The input layer is formed exactly like the input of the simple RNN layer in the previous section.

### Forget gate

The forget gate computes the forgetting factor

$$
f_t^i = \sigma \big( \mathbf W_f(i,:)\, \mathbf h_{t-1} + \mathbf U_f(i,:)\, \mathbf x_t + b_f^i \big)
$$

Like the input gate factor, it lies between 0 and 1. It sets how much of the earlier cell state is preserved.

### Output gate

The output gate *protects the subsequent cells* from a hidden state that is irrelevant to their context.

$$
q_t^i = \sigma \big( \mathbf W_o(i,:)\, \mathbf h_{t-1} + \mathbf U_o(i,:)\, \mathbf x_t + b_o^i \big)
$$

The output gate learns when to let the hidden state out.

### Cell state

The cell state is the core of the LSTM cell. It is the new memory that the LSTM introduces, and the gate factors above exist to preserve it for as long as the task needs it. The update is

$$
\begin{aligned}
a_t^i &= \mathbf W(i,:)\, \mathbf h_{t-1} + \mathbf U(i,:)\, \mathbf x_t + b^i \\
s_t^i &= f_t^i\, s_{t-1}^i + g_t^i \tanh ( a_t^i )
\end{aligned}
$$

The parameters $\boldsymbol\theta_{in} = \{ \mathbf W, \mathbf U, \mathbf b \}$ are the recurrent weights, the input weights and the bias at the input of the cells.

### Hidden state

The hidden state is the output of the LSTM cell. It depends on the cell state and the output gate:

$$
h_t^i = q_t^i \tanh(s_t^i)
$$

If the input and output gates are both closed (equal to 0) and the forget gate is fully open (equal to 1), the activation is trapped in the memory cell. It neither grows nor shrinks, and it does not affect the output at intermediate time steps.

### Output layer

The output layer is the same as in the simple RNN. Depending on the task (classification, regression and so on), it is typically a fully connected layer followed by a softmax or another output unit.

## Backward pass

Backpropagation works as in the simple RNN, with more complicated expressions. The additive update of the cell state forms what is called the *constant error carousel*. It lets the gradient flow back across many time steps without exploding or vanishing. In this sense the gates learn when to let the error in and when to let it out. In practice, LSTMs learn long-range dependencies much better than simple RNNs. The time series section that follows trains an LSTM and compares it with simple RNNs on the same data.

Greff et al. study the LSTM design and its hyperparameters systematically in ["LSTM: A Search Space Odyssey"](https://arxiv.org/pdf/1503.04069v1.pdf).

## Additional resources

These tutorials cover LSTMs from other angles:

1. [A Critical Review of Recurrent Neural Networks for Sequence Learning](https://arxiv.org/pdf/1506.00019.pdf)
2. [Understanding LSTMs](https://colah.github.io/posts/2015-08-Understanding-LSTMs)
3. [Illustrated guide to LSTMs](https://towardsdatascience.com/illustrated-guide-to-lstms-and-gru-s-a-step-by-step-explanation-44e9eb85bf21)
4. [Simplest possible LSTM explanation video](https://www.youtube.com/watch?v=WCUNPb-5EYI)

## PyTorch reference

| PyTorch class | Description |
| - | - |
| [`nn.LSTM`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.LSTM.html) | Multi-layer long short-term memory RNN. |
| [`nn.LSTMCell`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.LSTMCell.html) | One LSTM cell. |
| [`nn.Linear`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Linear.html) | Affine transformation $y = xA^T + b$. |

Key references: (Sak et al., 2014; Collins et al., 2016; Sutskever et al., 2014; Zaremba et al., 2014; Jaderberg et al., 2016)

## References

* Collins, J., Sohl-Dickstein, J., Sussillo, D. (2016). *Capacity and Trainability in Recurrent Neural Networks*. arXiv \[stat.ML]. [Link](http://arxiv.org/abs/1611.09913)
* Jaderberg, M., Czarnecki, W., Osindero, S., Vinyals, O., Graves, A., et al. (2016). *Decoupled Neural Interfaces using Synthetic Gradients*. arXiv \[cs.LG]. [Link](http://arxiv.org/abs/1608.05343)
* Sak, H., Senior, A., Beaufays, F. (2014). *Long Short-Term Memory Based Recurrent Neural Network Architectures for Large Vocabulary Speech Recognition*. arXiv \[cs.NE]. [Link](http://arxiv.org/abs/1402.1128)
* Sutskever, I., Vinyals, O., Le, Q. (2014). *Sequence to Sequence Learning with Neural Networks*. arXiv \[cs.CL]. [Link](http://arxiv.org/abs/1409.3215)
* Zaremba, W., Sutskever, I., Vinyals, O. (2014). *Recurrent Neural Network Regularization*. arXiv \[cs.NE]. [Link](http://arxiv.org/abs/1409.2329)

***

Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/lectures/rnn/lstm).


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.