> ## Documentation Index > Fetch the complete documentation index at: https://aegean.ai/llms.txt > Use this file to discover all available pages before exploring further. # Visual Instruction Tuning - LLaVA > Large language and vision assistant with visual instruction tuning. LLaVa (Large Language and Vision Assistant) is a state-of-the-art Visual Language Model (VLM) that excels in understanding and generating responses based on visual inputs. It builds upon the foundation of large language models by incorporating visual instruction tuning, enabling it to interpret images and provide contextually relevant answers. LLaVa leverages a combination of pre-trained vision encoders and large language models, fine-tuned on a diverse set of multimodal datasets to enhance its ability to follow visual instructions effectively. This makes LLaVa particularly adept at tasks such as image captioning, visual question answering, and other applications that require a deep understanding of both visual and textual information.
Visual Instruction Tuning, LLaVA

Visual Instruction Tuning (LLaVA)

Liu et al., arXiv:2304.08485 (PDF)

Improved Baselines with Visual Instruction Tuning (LLaVA-1.5)

Liu et al., arXiv:2310.03744, updated version of LLaVA with improved visual encoder and MLP connector

LLaVA section

Open in Google Colab

LLaVA, Hugging Face Transformers

Model documentation and API reference on Hugging Face

 **Key references**: (Xu et al., 2015; Johnson et al., 2016; Lu et al., 2016; Vinyals et al., 2016; Anderson et al., 2017) ## LLaVA fine-tuning The original LLaVA paper introduced the **LLaVA-Instruct-150K** dataset to fine-tune the model on visual instruction following. It contains approximately 150K GPT-generated instruction–answer pairs grounded in COCO images, formatted as multi-turn conversations. Each sample pairs a COCO image with a conversation between a human and a GPT assistant: ```json theme={null} { "image": "COCO_train2014_000000000009.jpg", "conversations": [ {"from": "human", "value": "What is the man doing?"}, {"from": "gpt", "value": "He is riding a skateboard."} ] } ``` **Dataset properties** | Property | Value | | -------- | ---------------------------------------- | | Size | \~150K samples | | Source | GPT-4 generated instructions and answers | | Format | Multi-turn conversational | | Images | COCO train2014 | This dataset can be used directly with the Hugging Face Transformers LLaVA pipeline for supervised fine-tuning.

LLaVA-Instruct-150K

Dataset on Hugging Face, liuhaotian/LLaVA-Instruct-150K

 ## PyTorch reference | PyTorch class | Description | | -------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------ | | [`nn.Conv2d`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Conv2d.html) | Applies a 2D convolution over an input signal composed of several input planes. | | [`nn.MultiheadAttention`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.MultiheadAttention.html) | Allows the model to jointly attend to information from different representation subspaces. | | [`nn.LayerNorm`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.LayerNorm.html) | Applies Layer Normalization over a mini-batch of inputs. | | [`nn.Linear`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Linear.html) | Applies an affine linear transformation to the incoming data: $y = xA^T + b$. | | [`nn.Embedding`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.Embedding.html) | A simple lookup table that stores embeddings of a fixed dictionary and size. | | [`nn.GELU`](https://docs.pytorch.org/docs/2.12/generated/torch.nn.GELU.html) | Applies the Gaussian Error Linear Units function. | ## References * Anderson, P., He, X., Buehler, C., Teney, D., Johnson, M., et al. (2017). *Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering*. * Johnson, J., Hariharan, B., Maaten, L., Fei-Fei, L., Lawrence Zitnick, C., et al. (2016). *CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning*. * Lu, J., Xiong, C., Parikh, D., Socher, R. (2016). *Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioning*. * Vinyals, O., Toshev, A., Bengio, S., Erhan, D. (2016). *Show and Tell: Lessons learned from the 2015 MSCOCO Image Captioning Challenge*. * Xu, K., Ba, J., Kiros, R., Cho, K., Courville, A., et al. (2015). *Show, Attend and Tell: Neural Image Caption Generation with Visual Attention*. *** [Edit this page on GitHub](https://github.com/aegean-ai/eaia/edit/main/src/aiml-common/lectures/vlm/llava/index.mdx) or [file an issue](https://github.com/aegean-ai/eaia/issues/new/choose).