Teaching a Small VLM to Read an Assembly Manual

​

Grading

​

The task

Annin Robotics publishes a 377-page manual for building its AR4 robot arm. Most of the manual is assembly steps, and each step pairs a bench photo with a short printed instruction such as “(6) M3x20 flat head screws”. A person following it keeps asking the same kinds of question: how many screws this step takes and which kind, which part goes in, which tool is needed. On the parts-list pages they ask for facts, for example how many of one screw the whole robot needs. Answering means reading an image and a question together, which is the input a vision-language model takes. You start from SmolVLM-256M-Instruct, a vision-language model small enough to fine-tune on a free GPU. You measure how well it answers 344 held-out questions about the manual without any training. Then you fine-tune it on 1,149 training questions in three ways that change very different parts of the model. LoRA adds small trainable low-rank matrices beside the frozen attention and MLP weights. DoRA separates each of those weights into a magnitude and a direction and adapts the direction. LayerNorm tuning trains only the normalization layers, about 35 thousand parameters out of 256 million. You compare what each method buys in accuracy against what it costs in parameters, memory, time and checkpoint size. The questions are in the aegean-ai/ar4-assembly-vqa dataset. It stores no images. Each row names a page of the manual and a box on that page, and the code you are given downloads the manual and cuts the images itself. Answers are scored by normalized match: both strings are lowercased, punctuation and the articles a, an and the are removed, and what remains must be the same words. Exact match is reported beside it.

​

What you hand in

• The notebook, with every cell output saved.
• The CSV and JSON files in its runs folder, one pair per method. Leave out the checkpoint folders.

​

What you will need

You need a GPU runtime. In Colab, open Runtime, select Change runtime type, and choose a T4 GPU before starting. On the A4000 used to test the assignment, LoRA trained in about 21 minutes, DoRA in about 34 and LayerNorm tuning in about 8, and scoring the test split took 2 to 3 minutes per method. A T4 is slower, so allow about two hours in total. A free Colab session may end before you finish. Every method saves its results to the runs folder as soon as it is scored, and a method whose files are already there is loaded instead of trained again. Download runs after each method and upload it again when you continue in a new session. You do not need a Hugging Face token. The model and the dataset are public.

​

Assignment notebook

​

Submitting

Commit the notebook and the runs files to your private clone of the class repo under assignments/assignment-3. Save every cell output before committing. See the assignment submission guide for more information.

---