> ## Documentation Index
> Fetch the complete documentation index at: https://aegean.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Vision-Language Models

> Vision-language models and multimodal AI systems, CLIP and LLaVA.

This chapter covers multimodal AI systems that combine vision and language. We start by presenting a generic view of the

* **CLIP**, the classic image-text alignment model that uses contrastive learning to embed images and captions into a shared space.
* **LLaVA**, directly combines a vision encoder (CLIP ViT) with a large language model (Llama 2, Vicuna) for instruction-following, dialogue, and vision-language reasoning.

## In this chapter

<CardGroup cols={2}>
  <Card title="CLIP" icon="link" href="/aiml-common/lectures/VLM/clip/index">
    Contrastive Language-Image Pretraining, the foundation of modern vision-language models.
  </Card>

  <Card title="CLIP Zero-Shot Classification" icon="wand-magic-sparkles" href="/aiml-common/lectures/VLM/clip/zero-shot/index">
    Hands-on section: zero-shot classification as a linear classifier whose weights come from text prompts.
  </Card>

  <Card title="LLaVA" icon="eye" href="/aiml-common/lectures/VLM/llava/index">
    Large Language and Vision Assistant, instruction-tuned multimodal dialogue.
  </Card>
</CardGroup>

**Key references**: (Lu et al., 2016; Johnson et al., 2016; Xu et al., 2015; Vinyals et al., 2016)

## References

* Johnson, J., Hariharan, B., Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., et al. (2016). *CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning*.
* Lu, J., Xiong, C., Parikh, D., Socher, R. (2016). *Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning*.
* Vinyals, O., Toshev, A., Bengio, S., Erhan, D. (2016). *Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge*.
* Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., et al. (2015). *Show, Attend and Tell: Neural Image Caption Generation with Visual Attention*.

***

<Callout icon="pen-to-square" iconType="regular">
  [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/vlm/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).
</Callout>