Skip to main content
The simplest RNN has a single layer of neurons that receive the input x\mathbf{x}. A single RNN layer unrolled over time, mapping each input to a hidden state Simple RNN layer with nneuronsn_{neurons} RNN neurons that map the input to the hidden state at each time step. A more practical simple RNN adds recurrences between the hidden units and an output at every step. An RNN with hidden-to-hidden recurrence and an output and loss at each time step, shown folded and unrolled Simple RNN with recurrences between hidden units. This architecture can compute any computable function, so it is as powerful as a Universal Turing Machine. Notice how the input xt−1\mathbf x_{t-1} affects the label yt\mathbf y_{t} through the hidden state. Notice also that the outputs y\mathbf y are conditionally independent of each other given the inputs x\mathbf x.
To use this diagram for backpropagation, swap the roles of its nodes and edges. In a computational graph the tensors sit on the edges and the RNN functions (gates) sit in the nodes.

Dimensioning simple RNNs

In the table below, mm is the number of examples in the mini-batch. You can stack several such layers on top of each other. Each layer keeps its own hidden state.

Forward propagation

This network maps an input sequence to an output sequence of the same length. Its forward pass is: at=Wht−1+Uxt+bht=tanh⁡(at)ot=Vht+cy^t=softmax(ot)\begin{aligned} \mathbf a_t &= \mathbf W \mathbf h_{t-1} + \mathbf U \mathbf x_t + \mathbf b \\ \mathbf h_t &= \tanh(\mathbf a_t) \\ \mathbf o_t &= \mathbf V \mathbf h_t + \mathbf c \\ \hat{\mathbf y}_t &= \mathtt{softmax}(\mathbf o_t) \end{aligned} The time series forecasting section implements the first two lines as a small PyTorch module and checks it against nn.RNN. The loss over a sequence of length τ\tau is the KL divergence between the empirical distribution of the data and the model distribution. Up to a constant that does not depend on θ\boldsymbol\theta, it is the negative log-likelihood: L=DKL[p^data(y∣x) ∥ pmodel(y∣x;θ)]=−Ey∣x∼p^datalog⁡pmodel(y∣x;θ)=−∑tlog⁡pmodel(yt∣x1,…,xt;θ)\begin{aligned} L &= D_{KL}\big[\hat p_{data}(\mathbf y | \mathbf x) \,\|\, p_{model}(\mathbf y | \mathbf x; \boldsymbol\theta)\big] \\ &= - \mathbb E_{\mathbf y | \mathbf x \sim \hat{p}_{data}} \log p_{model}(\mathbf y | \mathbf x ; \boldsymbol\theta) \\ &= - \sum_t \log p_{model}(y_t | \mathbf x_1, \dots, \mathbf x_t ; \boldsymbol\theta) \end{aligned} where LL is shorthand for L(y^1,…,y^τ,y1,…,yτ)L(\hat{\mathbf y}_1, \dots , \hat{\mathbf y}_{\tau}, \mathbf y_1, \dots , \mathbf y_{\tau}). RNNs can model very general distributions log⁡pmodel(x,y;θ)\log p_{model}(\mathbf x, \mathbf y ; \boldsymbol\theta). The simple RNN above models the conditional distribution log⁡pmodel(y∣x;θ)\log p_{model}(\mathbf y | \mathbf x ; \boldsymbol\theta). Because of the conditional independence assumption, that distribution factorizes into ∑tlog⁡pmodel(yt∣x1,…,xt;θ)\sum_t \log p_{model}(y_t | \mathbf x_1, \dots, \mathbf x_t ; \boldsymbol\theta).
You can remove this simplifying assumption by connecting yt−1\mathbf y_{t-1} to ht\mathbf h_t through another matrix, for example R\mathbf R. The network can then model an arbitrary distribution log⁡pmodel(y∣x;θ)\log p_{model}(\mathbf y | \mathbf x ; \boldsymbol\theta). As in other deep network architectures, the connectivity directly affects the representational capacity of the hypothesis set.
In many problems only the label yτy_\tau at the end of the sequence matters. Suppose you classify speech or video from inside a car cabin to detect the psychological state of the driver. The same architecture covers this case: you keep only oτ\mathbf o_\tau, LτL_\tau and yτy_\tau and ignore the outputs at earlier steps. An example makes the forward equations concrete. A six-token sentence fed one token at a time into an unrolled RNN Example sentence as input to the RNN The figure shows a short document (a sentence) broken into tokens. Here a token is a word, and the sentence has τ=6\tau=6 tokens. Suppose you need one label for the whole document. At t=1t=1 the network receives the first token x1\mathbf x_1. Starting from the hidden state h0=0\mathbf h_0 = 0, it computes h1\mathbf h_1 with the forward equations and ignores the output o1\mathbf o_1. It repeats this unrolling for each new input x2,x3,…\mathbf x_2, \mathbf x_3, \dots until it reaches the end-of-sentence token x6\mathbf x_6. Only at that step does it compute the output y6y_6 and the loss −log⁡pmodel(y6∣x1,…,x6;θ)- \log p_{model} (y_6|\mathbf x_1, \dots , \mathbf x_6; \boldsymbol\theta) where θ={W,U,V,b,c}\boldsymbol\theta = \{ \mathbf W, \mathbf U, \mathbf V, \mathbf b, \mathbf c \}.

Backpropagation through time (BPTT)

Training uses backpropagation on the unrolled network. Computational graph of two unrolled recurrences, t minus 1 and t, with the backward gradient flow Understanding RNN memory through the BPTT procedure Backpropagation in an RNN looks like backpropagation in other networks, because the unrolled architecture resembles a feed-forward one. There is one important difference, which the computational graph above shows for recurrences t−1t-1 and tt. The computation of ht\mathbf h_t uses the value ht−1\mathbf h_{t-1} from the previous recurrence. When you apply the chain rule in the backward pass, every node that involves such a recurrent variable receives a non-local gradient from a later step (tt in the figure). In the networks you have seen so far, each gate only used gradients local to it. This is why a simple RNN is said to have memory. Look at the backward pass in recurrence t−1t-1, at the junction between tanh⁡\tanh and Vht−1\mathbf V \mathbf h_{t-1}. The gradient ∇ht−1Lt\nabla_{\mathbf h_{t-1}}L_t arrives there from the backpropagation of the Wht−1\mathbf W \mathbf h_{t-1} node in recurrence tt. Because the node is a junction, this gradient is added to the gradient that flows down from the output of recurrence t−1t-1: ∇ht−1Lt−1←∇ht−1Lt−1+∇ht−1Lt\nabla_{\mathbf h_{t-1}}L_{t-1} \leftarrow \nabla_{\mathbf h_{t-1}}L_{t-1} + \nabla_{\mathbf h_{t-1}}L_t Section 10.2.2 of Goodfellow, Bengio and Courville gives the exact equations. The intuition behind the computational graph is enough to follow the rest of this chapter. In practice BPTT is truncated. Without truncation, a single gradient update would need a full forward pass and a full backward pass over the training sequence, which for a language model is very long. If the truncation length is too short, the bias it introduces can stop SGD from converging. The truncation length is usually chosen by a heuristic, larger than the expected memory of the system, or by cross-validation. PyTorch runs BPTT for you: calling backward() on a loss computed from an unrolled nn.RNN sends the gradient back through every step. The time series forecasting section trains simple and deep RNNs this way to predict the next values of a series.

Vanishing or exploding gradients

The figure below is a conceptual picture of a recurrence over time. It is called an infinite impulse response (IIR) filter, for reasons that become clear shortly.A first-order IIR filter: the input is added to the delayed output scaled by a weight wInfinite impulse response (IIR) filter with weight wwLet DD denote a unit delay, a memory location that stores a value for one step. The recurrence of this system isht=wht−1+xth_t = w h_{t-1} + x_twhere ww is a scalar weight. Feed an impulse xt=δtx_t = \delta_t into the system with w=−0.9w=-0.9:h0=−0.9 h−1+δ0=1h1=−0.9 h0+δ1=−0.9h2=−0.9 h1+δ2=0.81h3=−0.9 h2+δ3=−0.729\begin{aligned} h_0 &= -0.9\, h_{-1} + \delta_0 = 1 \\ h_1 &= -0.9\, h_{0} + \delta_1 = -0.9 \\ h_2 &= -0.9\, h_{1} + \delta_2 = 0.81 \\ h_3 &= -0.9\, h_{2} + \delta_3 = -0.729 \end{aligned}With w=−0.9w=-0.9, the impulse response hth_t follows a decaying exponential envelope. With ∣w∣>1|w| > 1 it would follow a growing exponential envelope. If the recurrence continues, the response vanishes or explodes long after the impulse entered at t=0t=0. In this simple filter, the weight alone decides which of the two happens.
The RNN hidden-state recurrence behaves the same way in the backward pass, which runs from t=τt=\tau back to t=1t=1. When τ\tau is large, the gradient can vanish or explode. The scalar ww of the IIR filter becomes the matrix W\mathbf W, and the signal hh becomes the gradient ∇htLt\nabla_{\mathbf h_{t}}L_{t}. Pascanu, Mikolov and Bengio analyze this in detail. The derivative of the tanh⁡\tanh non-linearity, shown below, lies between 0 and 1. It shrinks the gradients and slows down training. Plot of the derivative of tanh, which peaks at 1 at the origin and decays toward 0 Derivative of the tanh⁡\tanh non-linearity The repeated multiplication by W\mathbf W has a similar effect. If you ignore the non-linearity, the hidden state is ht=Wht−1\mathbf h_{t} = \mathbf W \mathbf h_{t-1}, so after τ\tau steps hτ=Wτh0\mathbf h_{\tau} = \mathbf W^\tau \mathbf h_{0} The same power of W\mathbf W appears in the expression for ∇htLt\nabla_{\mathbf h_{t}}L_{t} (see section 10.2.2). If the magnitudes of the eigenvalues of W\mathbf W are below 1, the gradients vanish. If they are above 1, the gradients explode. The long short-term memory (LSTM) architecture in the next section addresses this problem.

PyTorch reference

Key references: (Graves et al., n.d.; Jaderberg et al., 2016; Parisotto et al., 2016)

References

  • Graves, A., Wayne, G., Danihelka, I. (n.d.). Neural Turing Machines. Link
  • Jaderberg, M., Czarnecki, W., Osindero, S., Vinyals, O., Graves, A., et al. (2016). Decoupled Neural Interfaces using Synthetic Gradients. arXiv [cs.LG]. Link
  • Parisotto, E., Mohamed, A., Singh, R., Li, L., Zhou, D., et al. (2016). Neuro-Symbolic Program Synthesis. arXiv [cs.AI]. DOI

Found an error on this page? Report an issue.