Skip to main content
Open In Colab

​
Zero-shot classification as a linear classifier (CLIP)

This section demonstrates that zero-shot classification with CLIP can be interpreted as a linear classifier whose weights are generated from text prompts. We follow the notation used on the CLIP lecture page: ℓ\boldsymbol\ell for images, t\mathbf{t} for text, fℓf_\ell and ftf_t for the two encoders, and zℓ=fℓ(ℓ)\mathbf{z}_\ell = f_\ell(\boldsymbol\ell), zt=ft(t)\mathbf{z}_t = f_t(\mathbf{t}) for the embeddings. You will:
  1. Derive the formulation
  2. Implement zero-shot classification
  3. Visualize the text classifier weights and the image in the shared embedding space

​
1. Derivation

CLIP learns two encoders (see the CLIP lecture for the contrastive training objective): zℓ=fℓ(ℓ),zt=ft(t)\mathbf{z}_\ell = f_\ell(\boldsymbol\ell), \quad \mathbf{z}_t = f_t(\mathbf{t}) where ℓ\boldsymbol\ell is an image, t\mathbf{t} is a text string, fℓf_\ell is the image encoder, and ftf_t is the text encoder. Both output unit vectors on the surface of the hypersphere Sdz\mathbb{S}^{d_z}. For zero-shot classification over classes y∈{a,b,c,…}y \in \lbrace a, b, c, \ldots\rbrace , we write a prompt ty\mathbf{t}^y for each class and embed it: zty=ft(ty)\mathbf{z}_t^y = f_t(\mathbf{t}^y) Given a query image ℓq\boldsymbol\ell^q with embedding zℓq=fℓ(ℓq)\mathbf{z}_\ell^q = f_\ell(\boldsymbol\ell^q), the prediction is the class whose text embedding has the largest dot product with the image embedding: y^=arg⁡max⁡y  (zℓq)⊤zty\hat{y} = \arg\max_y \; (\mathbf{z}_\ell^q)^\top \mathbf{z}_t^y This is exactly a linear classifier over zℓq\mathbf{z}_\ell^q whose weight vectors are the per-class text embeddings: y^=arg⁡max⁡y  (zty)⊤zℓq\hat{y} = \arg\max_y \; (\mathbf{z}_t^y)^\top \mathbf{z}_\ell^q
Each text prompt ty\mathbf{t}^y instantiates one classifier weight vector zty\mathbf{z}_t^y. There is no training, the classifier is built on the fly from language.

​
2. Load a query image ℓq\boldsymbol\ell^q

Output from cell 4

​
3. Define prompts ty\mathbf{t}^y (classifier weights)

​
4. Compute the embeddings zℓq\mathbf{z}_\ell^q and {zty}\lbrace \mathbf{z}_t^y\rbrace

​
5. Zero-shot classification

Apply the classification rule from Section 1: y^=arg⁡max⁡y  (zℓq)⊤zty\hat{y} = \arg\max_y \; (\mathbf{z}_\ell^q)^\top \mathbf{z}_t^y

​
Why raw softmax looks flat

CLIP cosine similarities live in a narrow range. Both encoders produce L2L_2-normalized embeddings on Sdz\mathbb{S}^{d_z}, and in practice the joint space is a tight cone, so raw dot products (zℓq)⊤zty(\mathbf{z}_\ell^q)^\top \mathbf{z}_t^y typically sit between 0.18 and 0.30 even for good matches. A softmax applied directly to such close values produces a nearly uniform distribution, which makes the classifier look much less confident than it actually is. The fix is the learned temperature parameter τ\tau that CLIP was trained with (the same τ\tau that appears in the InfoNCE loss on the CLIP lecture page). In Stable-Baselines3’s HuggingFace wrapper this is stored as logit_scale = 1/τ ≈ 100. Inference must scale the similarities by 1/τ1/\tau before the softmax: p(y∣ℓq)=softmaxy ⁣(1τ⋅(zℓq)⊤zty)p(y \mid \boldsymbol\ell^q) = \mathrm{softmax}_y\!\left(\frac{1}{\tau} \cdot (\mathbf{z}_\ell^q)^\top \mathbf{z}_t^y\right) HuggingFace’s CLIP model exposes this scaled value as outputs.logits_per_image, so you get the sharp classification output for free. The cell below compares all three: raw cosine similarity, naive softmax (wrong), temperature-scaled softmax (correct), and the HuggingFace convenience.

​
6. Visualizing the classifier weights and the image in the same space

Plotting the first 50 components of each text embedding side by side is not very informative, the embeddings live on a dzd_z-dimensional unit hypersphere and the raw coordinates have no intrinsic meaning. A better view is to project the weights into a low-dimensional subspace and see how they are laid out relative to each other and to the image embedding zℓq\mathbf{z}_\ell^q. Below we fit a 2-component PCA to the text embeddings {zty}\lbrace \mathbf{z}_t^y\rbrace and project both the text prompts and the image into the same plane. Each labeled point is one classifier weight; the red star is the image embedding zℓq\mathbf{z}_\ell^q. The closest label to the star is the zero-shot prediction y^\hat{y}.
Output from cell 8

​
7. Conclusion

We demonstrated that:
  • Each class prompt ty\mathbf{t}^y produces a vector zty=ft(ty)\mathbf{z}_t^y = f_t(\mathbf{t}^y)
  • These vectors act as classifier weights
  • Zero-shot classification is linear classification in the shared CLIP embedding space
y^=arg⁡max⁡y  (zty)⊤zℓq\hat{y} = \arg\max_y \; (\mathbf{z}_t^y)^\top \mathbf{z}_\ell^q
Edit this page on GitHub or file an issue.