Skip to main content

Introduction

In probabilistic graphical modeling we saw the necessity of latent representations. In this section we advance the topic to the case where we need to learn features that are suitable for generating data from a complex data distribution. After reviewing the premise of generative modeling via what we will call deep latent variable models, we will look at the problem of inference of the latent space that is suitable for generating such data. We will see that this problem is intractable and we will introduce the variational inference method that will allow us to approximate the posterior distribution of the latent space. The method that we will dive into is called Variational Autoencoder (VAE) that is used across various domains, including collaborative filtering, image compression, reinforcement learning, and generation of content such as music and images. VAEs can also serve as the base to understand advanced topics such as Stable Diffusion Models (SDMs) that are used in the generation of images and videos. We will focus on continuous VAEs in this edition of the course.

Calculus of Variations

One of the main methods of generative approximate inference is variational inference that originates from calculus of variations. To understand how this calculus differs from the one we are used to, consider how the standard calculus using the derivative of a function is able to optimize it and calculate its minimum. In a similar way, in calculus of variations we are able to optimize a functional. A functional in mathematics is a mapping from a function space to the real numbers - just like the function is a mapping from a real number (the argument) to a real value (the result of the function evaluation). The concept of a functional can be illustrated with the definition of entropy, in the context of information theory. Shannon entropy, denoted as HH, is a measure of the uncertainty or randomness in a probability distribution. It’s defined for a discrete probability distribution P=(p1,p2,…,pn)P = (p_1, p_2, \ldots, p_n), where pip_i is the probability of the ii-th outcome. The entropy is defined as: H(P)=−∑i=1npilog⁡piH(P) = -\sum_{i=1}^{n} p_i \log p_i Here, HH is a functional because it maps the function (in this case, the probability distribution function PP) to a real number, which represents the entropy of PP. The calculus of variations is concerned with finding the function that maximizes a functional. In the case of entropy, the function that maximizes the entropy is the uniform distribution, where all the outcomes have the same probability. This is intuitive, because if all the outcomes have the same probability, then we have the most uncertainty about the outcome.

The Variational Generative Modeling Problem

VAE is one of the answers to the aforementioned variational optimization problem. In this approach, we use a neural network to create the right structure instead of explicitly encoding it in the prior. This is not a new idea: sampling from a complex probability distribution can be achieved by feeding samples from a simple distribution to a suitably chosen function. The conditional generator section builds this on a small example. A one-dimensional Gaussian latent variable is mapped to an angle and then to a point on a half-circle, and Gaussian noise turns the curve into a half-moon distribution in the plane. The neural network in a deep latent variable model plays the role of that function. With the function replaced by a neural network, we can now see a path to implementing the aforementioned generator, p(x∣θ)=∫p(x∣z;θ)p(z∣θ) dz p(\mathbf x | \mathbf \theta) = \int p(\mathbf x| \mathbf z ; \mathbf \theta) p(\mathbf z | \theta) \, d\mathbf z by assuming that the conditional distribution p(x∣z;θ)p(\mathbf x| \mathbf z ; \mathbf \theta) is parameterized by a neural network that takes as input the latent vector z\mathbf z generated itself by the prior distribution and outputs the image x\mathbf x. In the case that the p(x∣z;θ)p(\mathbf x| \mathbf z ; \mathbf \theta) is Gaussian, this will mean: p(x∣z;θ)=N(x;μ(z;θ),σ2I) p(\mathbf x| \mathbf z ; \mathbf \theta) = N(\mathbf x; \mu(\mathbf z; \mathbf \theta), \sigma^2 \mathbf I) where μ(z;θ)\mu(\mathbf z; \mathbf \theta) is the output of the neural network that is parameterized by θ\mathbf \theta. Note that the variance of the Gaussian is fixed, is a hyperparameter and is not a function of z\mathbf z. Note that the p(x∣z;θ)p(\mathbf x| \mathbf z ; \mathbf \theta) can be any distribution and obviously this distribution needs to match the nature of what we try to generate. If x\mathbf x is binary then this can be a Bernoulli distribution with the parameter(s) of the distribution given at the output of the neural network.

The Optimization Problem and Objective

The generator has parameters θ\mathbf \theta, the weights of the neural network. You choose them by maximum likelihood: you maximize the log marginal likelihood log⁡p(x∣θ)\log p(\mathbf x | \mathbf \theta) of the training data over θ\mathbf \theta. This is the quantity the optimization needs, and it is the quantity you cannot compute directly. The marginal p(x∣θ)p(\mathbf x | \mathbf \theta) integrates over every latent vector z\mathbf z. Even with neural networks “designing” the right features in the latent space 1, estimating this integral is a heavy computational problem. To see why, consider the MNIST dataset and the problem of generating handwritten digits that look like it. You can sample from p(z∣θ)p(\mathbf z | \theta), generating a large number of samples z1,…,zm{z_1, \dots , z_m}, and with the help of the neural network compute p(x)≈1m∑ip(x∣zi)p(\mathbf x) \approx \frac{1}{m} \sum_i p(\mathbf x|z_i). The problem is that you need a very large number of such samples in high dimensional spaces such as images (for MNIST this is 28×28 dimensions). Most of the samples zi\mathbf z_i will result into negligible p(x∣zi)p(\mathbf x|z_i) and therefore won’t contribute to the estimate of p(x)p(\mathbf x). See the geometry of the data distribution section for details. This is the problem that VAE addresses. The key idea behind its design is inference of the right latent space: you learn a distribution over z\mathbf z that concentrates on the values likely to have produced each training example x\mathbf x. When z\mathbf z is sampled from it, the result is an x\mathbf x similar to the training data. The VAE then optimizes a tractable lower bound on log⁡p(x∣θ)\log p(\mathbf x | \mathbf \theta) in place of the intractable quantity itself, over both the generator parameters and the parameters of this distribution. It gives you sample efficiency, so the objective can be computed and optimized with far less effort than before.

References

[1]: Diederik Kingma and Max Welling. Auto-Encoding Variational Bayes. In International Conference on Learning Representations, 2014, https://arxiv.org/abs/1312.6114 [2]: Yuri Burda, Roger Grosse, Ruslan Salakhutdinov. Importance Weighted Autoencoders. In International Conference on Learning Representations, 2015, https://arxiv.org/abs/1509.00519 [3]: Diederik P. Kingma, Max Welling. An Introduction to Variational Autoencoders. Foundations and Trends in Machine Learning, Vol. 12, No. 4, pp. 307-392, 2019, https://arxiv.org/abs/1906.02691
Found an error on this page? Report an issue.

Footnotes

  1. Note that the features that the neural network captures are not interpretable as the intuitively understood features that humans consider. For the MNIST dataset for example, humans will consider the slant of each digit, thinner strokes etc. ↩