> ## Documentation Index > Fetch the complete documentation index at: https://aegean.ai/llms.txt > Use this file to discover all available pages before exploring further. # Teaching a Small VLM to Read an Assembly Manual > Fine-tune SmolVLM-256M on questions about building the AR4 robot arm, and compare LoRA, DoRA and LayerNorm tuning. ## Grading | Component | Points | | - | - | | Part 1: the zero-shot baseline, `by_qtype`, `failure_kind` and question 1 | 20 | | Part 2: LoRA, DoRA and LayerNorm tuning trained and scored on the test split | 50 | | Part 3: questions 2 to 4, on the method comparison, the failure cases and leakage | 30 | | **Total** | **100** | ## The task Annin Robotics publishes a 377-page manual for building its AR4 robot arm. Most of the manual is assembly steps, and each step pairs a bench photo with a short printed instruction such as "(6) M3x20 flat head screws". A person following it keeps asking the same kinds of question: how many screws this step takes and which kind, which part goes in, which tool is needed. On the parts-list pages they ask for facts, for example how many of one screw the whole robot needs. Answering means reading an image and a question together, which is the input a vision-language model takes. You start from [SmolVLM-256M-Instruct](https://arxiv.org/abs/2504.05299), a vision-language model small enough to fine-tune on a free GPU. You measure how well it answers 344 held-out questions about the manual without any training. Then you fine-tune it on 1,149 training questions in three ways that change very different parts of the model. [LoRA](https://arxiv.org/abs/2106.09685) adds small trainable low-rank matrices beside the frozen attention and MLP weights. [DoRA](https://arxiv.org/abs/2402.09353) separates each of those weights into a magnitude and a direction and adapts the direction. [LayerNorm tuning](https://arxiv.org/abs/2312.11420) trains only the normalization layers, about 35 thousand parameters out of 256 million. You compare what each method buys in accuracy against what it costs in parameters, memory, time and checkpoint size. The questions are in the [`aegean-ai/ar4-assembly-vqa`](https://huggingface.co/datasets/aegean-ai/ar4-assembly-vqa) dataset. It stores no images. Each row names a page of the manual and a box on that page, and the code you are given downloads the manual and cuts the images itself. Answers are scored by normalized match: both strings are lowercased, punctuation and the articles a, an and the are removed, and what remains must be the same words. Exact match is reported beside it. ## What you hand in * The notebook, with every cell output saved. * The CSV and JSON files in its `runs` folder, one pair per method. Leave out the checkpoint folders. ## What you will need You need a GPU runtime. In Colab, open Runtime, select Change runtime type, and choose a T4 GPU before starting. On the A4000 used to test the assignment, LoRA trained in about 21 minutes, DoRA in about 34 and LayerNorm tuning in about 8, and scoring the test split took 2 to 3 minutes per method. A T4 is slower, so allow about two hours in total. A free Colab session may end before you finish. Every method saves its results to the `runs` folder as soon as it is scored, and a method whose files are already there is loaded instead of trained again. Download `runs` after each method and upload it again when you continue in a new session. You do not need a Hugging Face token. The model and the dataset are public. ### Assignment notebook > **[Open Notebook in Colab](https://colab.research.google.com/github/pantelis/eng-ai-agents/blob/main/notebooks/assignments/topics/vlm-finetuning/ar4-assembly-vqa/ar4-assembly-vqa.ipynb)** > > AR4 Assembly VQA Notebook ## Submitting Commit the notebook and the `runs` files to your private clone of the [class repo](https://github.com/pantelis/eng-ai-agents) under `assignments/assignment-3`. Save every cell output before committing. See the [assignment submission guide](https://aegean.ai/aiml-common/resources/environment/assignment-submission.md) for more information. *** Found an error on this page? [Report an issue](https://github.com/aegean-ai/eaia-feedback/issues/new?template=page-issue.yml\&page=/aiml-common/assignments/main/cv-fall-2026/assignment-3). This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.