> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.
# Language Models
> Introduction to statistical and neural language models for text generation.
These notes heavily borrow from [the CS224N set of notes on Language Models](https://web.stanford.edu/class/archive/cs/cs224n/cs224n.1224/index.html#coursework).
## The need for neural language models
\*Language model example that completes the search query. We need a language model that given a sequence of words ${ \mathbf w_1, ..., \mathbf w_t }$ it returns
$p(\mathbf w_{t+1} | \mathbf w_1, ..., \mathbf w_t)$
Language models in general compute the probability of occurrence of a number of words in a particular sequence. How we can build language models though ?
The probability of a sequence of $m$ words ${\mathbf w_1, ..., \mathbf w_m }$ is denoted as $p(\mathbf w_1,...,\mathbf w_m)$. For example the probability of the sentence "The cat sat on the mat" is $p(\text{The}, \text{cat}, \text{sat}, \text{on}, \text{the}, \text{mat})$.
We can simply use the chain rule of probability to compute the probability of a sequence of words without making any assumptions about the structure of the language. For example, the probability of the sentence "The cat sat on the mat" is:
$p(\text{The}, \text{cat}, \text{sat}, \text{on}, \text{the}, \text{mat}) = p(\text{The}) p(\text{cat} | \text{The}) p(\text{sat} | \text{The}, \text{cat}) p(\text{on} | \text{cat}, \text{sat}) p(\text{the} | \text{sat}, \text{on}) p(\text{mat} | \text{on}, \text{the})$
In general,
$ p(\mathbf w_1,...,\mathbf w_m) = \prod_{i=1}^{i=m} p(\mathbf w_{i} | \mathbf w_1, ..., \mathbf w_{i-1})$
Since the counting of all these probabilities is intractable, we will use a simpler model that makes some assumptions about the structure of the language. The assumption we usually make is that the probability of a word depends only on a fixed number of previous words. For example, we can assume that the probability of a word depends only on the previous n-words. This assumption is called the n-gram assumption.
$ p(\mathbf w_1,...,\mathbf w_m) \approx \prod_{i=1}^{i=m} p(\mathbf w_{i} | \mathbf w_{i-1}, ..., \mathbf w_{i-n+1})$
To compute these probabilities, the count of each n-gram would be compared against the frequency of each word. For instance, if the model takes bi-grams, the frequency of each bi-gram, calculated via combining a word with its previous word, would be divided by the frequency of the corresponding unigram. For example for bi-gram and trigram models the above equation becomes:
$p(\mathbf w_2 | \mathbf w_1) = \dfrac {count(\mathbf w_1,\mathbf w_2)}{count(\mathbf w_1)}$
$p(\mathbf w_3 | \mathbf w_1, \mathbf w_2) = \dfrac {count(\mathbf w_1, \mathbf w_2, \mathbf w_3)}{count(\mathbf w_1, \mathbf w_2)}$
Counting these probabilities is notoriously space (memory) inefficient as we need to store the counts for all possible grams in the corpus. At the same time, when we have no events in the denominators to count we have no possibility of estimating such probabilities (the so-called sparsity problem).
To solve these issues we will create neural models that are able to predict the required word.
## PyTorch reference
| PyTorch class | Description |
| -------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------- |
| [`nn.Embedding`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Embedding.html) | A simple lookup table that stores embeddings of a fixed dictionary and size. |
| [`nn.Linear`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Linear.html) | Applies an affine linear transformation to the incoming data: $y = xA^T + b$. |
| [`nn.Softmax`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Softmax.html) | Applies the Softmax function to an n-dimensional input Tensor. |
**Key references**: (Kim et al., 2015; Cho et al., 2014; Vinyals et al., 2015; Mikolov et al., 2013; Wiseman & Rush, 2016)
## References
* Cho, K., Merrienboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., et al. (2014). *Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation*.
* Kim, Y., Jernite, Y., Sontag, D., Rush, A. (2015). *Character-Aware Neural Language Models*.
* Mikolov, T., Chen, K., Corrado, G., Dean, J. (2013). *Efficient Estimation of Word Representations in Vector Space*.
* Vinyals, O., Fortunato, M., Jaitly, N. (2015). *Pointer Networks*.
* Wiseman, S., Rush, A. (2016). *Sequence-to-Sequence Learning as Beam-Search Optimization*.
***
[Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/nlp/language-models/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).