> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Multilayer Perceptron (MLP)
Notice that at its core the output of the multihead self attention is a weighted sum of the input tokens. This is a **linear combination** of the value vectors and the attention block does not have the capacity to learn non-linear relationships. One cat argue that the attention weights and softmax add some non-linearity but this is not enough and the model will not be able to learn complex dependencies. To address this we add an MLP to the output of the multihead self attention.
$Y = \mathtt{LayerNorm}(\hat Z)$
$\tilde X = \mathtt{MLP}(Y) + \hat Z$
where the MLP uses skip connection and for some implementations a RELU or GELU activation function shown below is used.
Notably, **each token is processed by the MLP independently of the other tokens in the input**.
```python theme={null}
class FeedForward(nn.Module):
def __init__(self, config):
super().__init__()
self.linear_1 = nn.Linear(config.hidden_size, config.intermediate_size)
self.linear_2 = nn.Linear(config.intermediate_size, config.hidden_size)
self.gelu = nn.GELU()
self.dropout = nn.Dropout(config.hidden_dropout_prob)
def forward(self, x):
x = self.linear_1(x)
x = self.gelu(x)
x = self.linear_2(x)
x = self.dropout(x)
return x
```
## Resources
1. [Dimensioning Transformers - Part 1 ](https://towardsdatascience.com/transformers-explained-visually-part-3-multi-head-attention-deep-dive-1c1ff1024853)
2. [Dimensioning Transformers - Part 2](https://towardsdatascience.com/transformers-explained-visually-part-2-how-it-works-step-by-step-b49fa4a64f34)
## PyTorch reference
| PyTorch class | Description |
| -------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
| [`nn.Linear`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Linear.html) | Applies an affine linear transformation to the incoming data: $y = xA^T + b$. |
| [`nn.GELU`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.GELU.html) | Applies the Gaussian Error Linear Units function. |
| [`nn.ReLU`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.ReLU.html) | Applies the rectified linear unit function element-wise. |
| [`nn.Dropout`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Dropout.html) | During training, randomly zeroes some of the elements of the input tensor with probability p. |
| [`nn.LayerNorm`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.LayerNorm.html) | Applies Layer Normalization over a mini-batch of inputs. |
**Key references**: (Tsai et al., 2019; Ramachandran et al., 2017; Chen et al., 2020; Dosovitskiy et al., 2020; Jetley et al., 2018)
## References
* Chen, T., Kornblith, S., Norouzi, M., Hinton, G. (2020). *A simple framework for contrastive learning of visual representations*.
* Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., et al. (2020). *An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale*.
* Jetley, S., Lord, N., Lee, N., Torr, P. (2018). *Learn To Pay Attention*.
* Ramachandran, P., Zoph, B., Le, Q. (2017). *Searching for Activation Functions*.
* Tsai, Y., Bai, S., Yamada, M., Morency, L., Salakhutdinov, R. (2019). *Transformer Dissection: A Unified Understanding of Transformer's Attention via the Lens of Kernel*.
***
[Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/nlp/transformers/mlp.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).