Getting Started on Linear Algebra

​

Grading

​

Nickname

Change your nickname in Discord server to your full name (first and last name). This is important for communication purposes.

​

Development Environment Setup

Please refer to the Development Environment Guide for setup instructions. After you clone or import the class github repo you need to provide proof that your development environment is correctly setup:

1. Complete this notebook to verify that Torch is running properly inside the docker container.. Note that you need to consult the repos Pull Requests if you are on a Apple M silicon platform.
2. Create an account on Hugging Face and provide a screenshot of your profile.

Note that screenshots must be stored as .png files and named dev-environment-docker-xyz.png and must be located under the assignment submission folder in your cloned class repo. If you have a very weak laptop and this does not allow you to launch dockers (highly unlikely), a screenshot of system monitor showing high CPU/RAM usage when trying to run a container as a proof that your laptop is too weak.

​

Low-Rank Gaussian Covariances

​

Assignment Notebook

Open Notebook in Colab

Low-Rank Gaussian Covariances Notebook

​

Learning objectives

By completing this assignment, you will:

1. Understand how matrix rank constrains changes in covariance structure.
2. Relate low-rank matrices to a small number of dominant variance directions.
3. Interpret eigenvalue spectra of covariance matrices.

The emphasis is conceptual and empirical rather than algebraic.

---

​

Background

Let $z \in \mathbb{R}^k$ be a latent Gaussian vector, $$z \sim \mathcal{N}(0, I_k),$$ and let $x \in \mathbb{R}^d$ be an observed vector, $$x = W z + \varepsilon, \qquad \varepsilon \sim \mathcal{N}(0, \sigma^2 I_d).$$ Then $x$ follows a multivariate Gaussian distribution $$x \sim \mathcal{N}(0, \Sigma), \qquad \Sigma = W W^\top + \sigma^2 I_d.$$ Throughout this notebook, you will reason about $\Sigma$ using eigenvalues and matrix rank.

---

​

Reference distribution

A reference coefficient matrix $W_0$ is given. It induces the reference covariance $$\Sigma_0 = W_0 W_0^\top + \sigma^2 I_d.$$ This covariance represents a baseline Gaussian distribution against which others will be compared.

---

​

Constructing Gaussian shifts

We now define two different ways to modify the coefficient matrix. Low-rank coefficient change A low-rank modification has the form $$\Delta W_\star = B A,$$ where:

• $B$ is a $d \times r$ matrix
• $A$ is an $r \times k$ matrix
• $r \ll \min(d,k)$

This construction limits the number of independent directions along which $W_0$ can change.

---

Diffuse coefficient change A diffuse modification is a dense matrix with approximately full rank. It is scaled so that the Frobenius norms are equal: $\|\Delta W_\mathrm{diffuse}\|_F = \|\Delta W_\mathrm{low-rank}\|_F$, ensuring both changes have the same overall magnitude.

---

​

Empirical covariance

From each modified coefficient matrix, samples are drawn and an empirical covariance $$S = \frac{1}{n} \sum_{i=1}^n x_i x_i^\top$$ is computed. These empirical covariances approximate the true Gaussian covariances.

---

​

Eigenvalue analysis

In this section, you will explicitly measure how different coefficient changes affect the spectrum of a Gaussian covariance matrix.

​

Step 1: Compute eigenvalues

For each of the following covariance matrices:

• the reference covariance $\Sigma_0$
• the empirical covariance from the low-rank modification $S_{\text{low-rank}}$
• the empirical covariance from the diffuse modification $S_{\text{diffuse}}$

compute all eigenvalues and sort them in decreasing order.

---

​

Step 2: Visualize the spectra

Create a single plot that shows:

• the eigenvalues of $\Sigma_0$
• the eigenvalues of $S_{\text{low-rank}}$
• the eigenvalues of $S_{\text{diffuse}}$

on the same axes. Use a logarithmic scale on the vertical axis. This plot should allow you to visually compare:

• how many eigenvalues are significantly larger than the noise floor,
• and how the different modifications affect the spectrum.

---

​

Step 3: Quantify “significant change”

Define an eigenvalue as significantly changed if it differs from the corresponding eigenvalue of $\Sigma_0$ by more than a fixed threshold (e.g., 10% relative difference). Using this criterion:

1. Count how many eigenvalues change significantly in the low-rank case.
2. Count how many eigenvalues change significantly in the diffuse case.

Report these numbers explicitly.

---

​

Step 4: Interpret the results

Answer the following questions in complete sentences:

1. In the low-rank case, how many eigenvalues change substantially, and how does this number relate to the rank of the coefficient modification?
2. In the diffuse case, why are many eigenvalues affected, even though the overall magnitude of the change is the same?
3. How do these observations illustrate the idea that matrix rank limits the number of variance directions that can change?

Your explanation should explicitly reference:

• eigenvalues,
• rank,
• and geometric directions of variance.

---