Backpropagation DNN exercises

Computational graph in Tensorboard showing the components involved in a TF BP update

Neuron

A network consist of a concatenation of the following layers

The task of backprop consists of the following steps:

Sketch the network and write down the equations for the forward path.
Propagate the backwards path i.e. make sure you write down the expressions of the gradient of the loss with respect to all the network parameters.

NOTE: Please note that we have omitted the bias terms for simplicity.

Backward Pass Step	Symbolic Equation
(5)	$\frac{\partial L}{\partial L} = 1.0$
(4)	$\frac{\partial L}{\partial z^{(2)}} = \hat y - y$
(3a)	$\frac{\partial L}{\partial W^{(2)}} = a^{(1)} (\hat y - y)$
(3b)	$\frac{\partial L}{\partial a^{(1)}} = W^{(2)} (\hat y - y)$
(2)	$\frac{\partial L}{\partial z^{(1)}} = \frac{\partial L}{\partial a^{(1)}}$ if $a^{(1)} > 0$
(1)	$\frac{\partial L}{\partial W^{(1)}} = \frac{\partial L}{\partial z^{(1)}} \times x^{(1)}$

Key references: (Choromanska et al., 2014; Romero et al., 2014; Bengio, 2012; Jaderberg et al., 2016; Bengio et al., 2015)

Bengio, Y. (2012). Practical recommendations for gradient-based training of deep architectures.
Bengio, E., Bacon, P., Pineau, J., Precup, D. (2015). Conditional Computation in Neural Networks for faster models.
Choromanska, A., Henaff, M., Mathieu, M., Ben Arous, G., LeCun, Y. (2014). The Loss Surfaces of Multilayer Networks.
Jaderberg, M., Czarnecki, W., Osindero, S., Vinyals, O., Graves, A., et al. (2016). Decoupled Neural Interfaces using Synthetic Gradients.
Romero, A., Ballas, N., Kahou, S., Chassang, A., Gatta, C., et al. (2014). FitNets: Hints for Thin Deep Nets.