Visual Tokens and Hallucination

The LLaVA deep dive described how a vision language model is assembled: an image encoder produces patch features, a connector turns them into tokens of the language model, and the language model reads them together with the text. This section measures two things that the architecture diagram does not show. The first is cost: how many tokens an image becomes, and how that number grows with the image size. The second is reliability: how to score a model’s answers, and how often a model claims to see objects that are not in the image. The measurements use SmolVLM2-500M, the model fine-tuned in the SmolVLM lab. It is small enough to run on one consumer GPU, and the same questions apply to every model of this kind.

import time

import numpy as np
import torch
from PIL import Image
from transformers import AutoModelForImageTextToText, AutoProcessor

device = "cuda" if torch.cuda.is_available() else "cpu"
MODEL_ID = "HuggingFaceTB/SmolVLM2-500M-Video-Instruct"
processor = AutoProcessor.from_pretrained(MODEL_ID)
model = AutoModelForImageTextToText.from_pretrained(MODEL_ID, dtype=torch.bfloat16).to(device).eval()
IMAGE_TOKEN_ID = processor.tokenizer.convert_tokens_to_ids("<image>")
print(f"[INFO] device: {device}, <image> token id: {IMAGE_TOKEN_ID}")

[INFO] device: cuda, <image> token id: 49190

Visual tokens

A language model reads a sequence of tokens, and an image has to become part of that sequence. In LLaVA-1.5 the image encoder is a ViT with

14 \times 14

patches on a

336 \times 336

input, so it returns

24 \times 24 = 576

patch features, and an MLP maps each one to a token: every image costs 576 tokens, the length of a page of text. Two choices change that number. Higher resolution helps the model read small text and see small objects, but the number of patches grows with the square of the side length. Merging neighboring patches before they enter the language model reduces the count. SmolVLM2 uses both. Its SigLIP encoder takes

512 \times 512

inputs with

16 \times 16

patches, giving

32 \times 32 = 1024

patch features. The connector merges each

4 \times 4

block of features into one vector (pixel shuffle) and projects it, so a

512 \times 512

input becomes 64 tokens. The processor first resizes every image so that its longer side is longest_edge pixels, 2048 by default, then cuts it into

512 \times 512

tiles. Each tile becomes 64 tokens, and a downscaled copy of the whole image adds 64 more, so that the model sees both the detail and the overall layout. The size of the original image therefore does not matter; the setting longest_edge does. The cell builds the same prompt for one photo at four values of longest_edge, and once with splitting turned off, counts the <image> tokens in the input, and times the prefill: the forward pass over the prompt that has to finish before the first answer token appears.

from datasets import load_dataset

pope = load_dataset("lmms-lab-encoder/POPE", "Full")
print(pope)
photo = pope["random"][0]["image"].convert("RGB")
print(f"[INFO] example photo: {photo.size[0]} x {photo.size[1]}")


def build_inputs(images, questions, split=True, longest_edge=2048):
    texts = []
    for q in questions:
        messages = [{"role": "user", "content": [{"type": "image"}, {"type": "text", "text": q}]}]
        texts.append(processor.apply_chat_template(messages, add_generation_prompt=True))
    return processor(text=texts, images=[[im] for im in images], return_tensors="pt", padding=True,
                     do_image_splitting=split, size={"longest_edge": longest_edge}
                     ).to(device, dtype=torch.bfloat16)


@torch.no_grad()
def prefill_ms(inputs, repeats=5):
    model(**inputs)  # warm-up
    torch.cuda.synchronize()
    start = time.perf_counter()
    for _ in range(repeats):
        model(**inputs)
    torch.cuda.synchronize()
    return 1000 * (time.perf_counter() - start) / repeats


rows = []
for edge, split in [(512, True), (1024, True), (1536, True), (2048, True), (2048, False)]:
    x = build_inputs([photo], ["Describe the image."], split=split, longest_edge=edge)
    n_img = int((x["input_ids"] == IMAGE_TOKEN_ID).sum())
    rows.append((edge, split, n_img, x["input_ids"].shape[1], prefill_ms(x)))

print(f"{'longest_edge':>12}  {'split':>5}  {'image tokens':>12}  {'prompt tokens':>13}  {'prefill ms':>10}")
for edge, split, n_img, n_all, ms in rows:
    print(f"{edge:>12}  {str(split):>5}  {n_img:>12}  {n_all:>13}  {ms:>10.1f}")

DatasetDict({
    adversarial: Dataset({
        features: ['id', 'question_id', 'question', 'answer', 'image_source', 'image', 'category'],
        num_rows: 3000
    })
    popular: Dataset({
        features: ['id', 'question_id', 'question', 'answer', 'image_source', 'image', 'category'],
        num_rows: 3000
    })
    random: Dataset({
        features: ['id', 'question_id', 'question', 'answer', 'image_source', 'image', 'category'],
        num_rows: 3000
    })
})
[INFO] example photo: 640 x 427

longest_edge  split  image tokens  prompt tokens  prefill ms
         512   True            64             79        29.9
        1024   True           320            345        32.1
        1536   True           448            477        34.1
        2048   True           832            874        42.5
        2048  False            64             79        30.2

The number of image tokens follows the tile grid. At longest_edge = 512 the whole photo fits in one tile and becomes 64 tokens. At 1024 the

640 \times 427

photo is resized to

1024 \times 683

pixels and cut into

2 \times 2

tiles, which with the global view gives

5 \times 64 = 320

tokens. At 1536 it is

3 \times 2

tiles and 448 tokens, and at the default of 2048 it is

4 \times 3

tiles and 832 tokens, about ten times the length of the text of the prompt. With splitting turned off, only the downscaled whole image is kept, 64 tokens. The prefill time grows much less than the token count, from 30 ms to 42.5 ms, because one short prompt on a large GPU leaves most of the GPU idle. With batches of images, longer conversations or a smaller GPU, the extra tokens cost proportionally more time and memory.

Scoring answers

A model answers in free text, and the score depends on how that text is compared with the reference. Three methods are common. Exact match compares the answer, after lower-casing and removing punctuation and articles, with the reference. It is strict: “two dogs” against the reference “2” counts as wrong. The VQA benchmarks soften it by collecting ten human answers per question and giving full credit when at least three annotators gave the model’s answer,

\text{acc} = \min(\#\text{matching humans}/3,\ 1)

. Benchmarks with multiple-choice questions, such as MMBench and MMMU, avoid the problem by asking for a letter. For open answers such as descriptions, a second, larger model can act as a judge and grade the answer against the reference. This scales, but the judge has its own errors and biases, and its scores are only comparable when the same judge and prompt are used. The training loss is not one of these measures. The SmolVLM lab reached 98% token accuracy on its validation set, because most tokens of each answer are the fixed JSON field names, which says little about whether the listed food items are right. Assignment 3 measures normalized exact match per question and classifies each failure, which is the kind of measurement the loss cannot replace.

Object hallucination

A vision language model hallucinates an object when it describes or confirms an object that is not in the image. The language model has learned which objects usually appear together, and that prior can outweigh the visual evidence: a model shown a dining table may report a fork because tables usually have forks. POPE (Polling-based Object Probing Evaluation) measures this with yes/no questions of the form Is there a \{object\} in the image? on COCO photos (arXiv:2305.10355). Half of the questions name an object that is in the image, and the answer is “yes”. The other half name an absent object, chosen in three ways, which gives three test sets of 3,000 questions each:

random: any COCO object category that is not in the image;
popular: the categories that appear most often in COCO overall;
adversarial: the categories that most often appear together with the objects that are in the image.

The adversarial set tests the language prior directly. If the model answers from co-occurrence instead of from the image, it fails most there. Besides accuracy, POPE reports precision, recall and F1 with “yes” as the positive class, and the fraction of “yes” answers. A model that is biased toward agreeing answers “yes” more than half of the time. The cell asks SmolVLM2 the first 500 questions of each set, with ” Answer yes or no.” appended, and reads the first word of the answer.

N_PER_SPLIT = 500
BATCH = 16
processor.tokenizer.padding_side = "left"   # generation continues from the right end of the prompt


@torch.no_grad()
def answer_yes_no(images, questions):
    x = build_inputs(images, [q + " Answer yes or no." for q in questions])
    out = model.generate(**x, max_new_tokens=3, do_sample=False)
    text = processor.batch_decode(out[:, x["input_ids"].shape[1]:], skip_special_tokens=True)
    return ["yes" if t.strip().lower().startswith("yes") else "no" for t in text]


def pope_scores(pred, truth):
    pred, truth = np.array(pred) == "yes", np.array(truth) == "yes"
    tp, fp = (pred & truth).sum(), (pred & ~truth).sum()
    fn = (~pred & truth).sum()
    precision, recall = tp / max(tp + fp, 1), tp / max(tp + fn, 1)
    return {"accuracy": (pred == truth).mean(), "precision": precision, "recall": recall,
            "F1": 2 * precision * recall / max(precision + recall, 1e-9), "yes ratio": pred.mean()}


results, examples = {}, {}
for split in ("random", "popular", "adversarial"):
    data = pope[split].select(range(N_PER_SPLIT))
    preds = []
    for i in range(0, N_PER_SPLIT, BATCH):
        batch = data[i:i + BATCH]
        preds += answer_yes_no([im.convert("RGB") for im in batch["image"]], batch["question"])
    truth = [a.strip().lower() for a in data["answer"]]
    results[split] = pope_scores(preds, truth)
    examples[split] = [(j, data[j]["question"]) for j in range(N_PER_SPLIT)
                       if preds[j] == "yes" and truth[j] == "no"][:3]

print(f"{'set':<12}" + "".join(f"{k:>11}" for k in results["random"]))
for split, r in results.items():
    print(f"{split:<12}" + "".join(f"{v:>11.3f}" for v in r.values()))

set            accuracy  precision     recall         F1  yes ratio
random            0.562      0.533      0.992      0.694      0.930
popular           0.528      0.515      0.992      0.678      0.964
adversarial       0.528      0.515      0.988      0.677      0.960

# Check the answers: the raw text for a few questions, and batched against one-at-a-time generation
data = pope["adversarial"].select(range(32))
images = [im.convert("RGB") for im in data["image"]]


@torch.no_grad()
def raw_answers(images, questions):
    x = build_inputs(images, [q + " Answer yes or no." for q in questions])
    out = model.generate(**x, max_new_tokens=3, do_sample=False)
    return processor.batch_decode(out[:, x["input_ids"].shape[1]:], skip_special_tokens=True)


batched = raw_answers(images[:16], data["question"][:16]) + raw_answers(images[16:], data["question"][16:])
single = [raw_answers([im], [q])[0] for im, q in zip(images, data["question"])]
for q, a, r in list(zip(data["question"], data["answer"], batched))[:6]:
    print(f"{q:<45} truth: {a:<4} model: {r!r}")
print(f"batched and one-at-a-time answers agree on {sum(b == s for b, s in zip(batched, single))} of 32")

Is there a snowboard in the image?            truth: yes  model: ' Yes.'
Is there a backpack in the image?             truth: no   model: ' Yes.'
Is there a person in the image?               truth: yes  model: ' Yes.'
Is there a car in the image?                  truth: no   model: ' Yes.'
Is there a skis in the image?                 truth: yes  model: ' Yes.'
Is there a dog in the image?                  truth: no   model: ' Yes.'
batched and one-at-a-time answers agree on 31 of 32

SmolVLM2-500M answers “yes” to 93% of the random questions and to 96% of the popular and adversarial ones. Its recall is therefore almost perfect (0.99): it rarely denies an object that is there. Its precision is barely above the 0.5 of a coin flip, so a “yes” carries almost no information, and the accuracy stays between 53% and 56%. The parsing is not the cause. The raw answers above show the model writing “Yes.” for the backpack, the car and the dog that are not in the photo, and batched generation agrees with one question at a time on 31 of 32 questions. In the figure, the model confirms a car and a backpack on a photo of a snowboarder alone on an empty slope. With a bias this strong, the three sets barely differ: accuracy drops from 56.2% on the random set to 52.8% on the other two, because the model says “yes” to almost everything regardless of which object is asked about. A model that always agrees would score 50% on every set. POPE is most informative for models whose “yes” ratio is near one half, where the gap between the random and the adversarial set measures how much the answers depend on the language prior.

Exercises

Repeat the POPE measurement with SmolVLM2-256M and with the model you fine-tuned in the SmolVLM lab. Does fine-tuning on food photos change the hallucination rate on COCO?
Run POPE with image splitting turned off. Does the lower number of image tokens change the accuracy on any of the three sets?
Change the question to Is there a \{object\} in the image? Answer yes only if you are sure. How do precision, recall and the yes ratio move?
Write a scorer for the food extraction task of the SmolVLM lab that reports, per image, whether the JSON parses, and the precision and recall of the listed food items against the label.

Key references

Li et al. introduced POPE and showed that object hallucination in vision language models follows the co-occurrence statistics of their training data (arXiv:2305.10355). Liu et al. describe the LLaVA-1.5 connector and its 576 visual tokens (Liu et al., 2023).

References

Liu, H., Li, C., Li, Y., Lee, Y. (2023). Improved baselines with visual instruction tuning. arXiv [cs.CV].

Found an error on this page? Report an issue.

Vision and Language

Vision Transformers

Representation Learning

Vision-Language Models

VLM SFT Lab

Prompted Segmentation

Visual Tokens and Hallucination

Visual tokens

Scoring answers

Object hallucination

Exercises

Key references

References