PaperfootJoin free
arXiv

Evaluation code shifts vision model scores on an image benchmark

A code choice for reading answers caused a 6.13-point adversarial score shift, exceeding the gains reported by five of six surveyed papers.

Paper by Kesav Kumar Jayakumar, Karthigeyan Thilak
Figure 2. Confusion matrices for POPE adversarial (3,000 questions) under each protocol.
Open the figure at full size
Figure 2. Confusion matrices for POPE adversarial (3,000 questions) under each protocol.Confusion matrices for POPE adversarial (3,000 questions) under each protocol.Jayakumar et al., CC BY
Preprint: empirical study

Edited by Boris Djordjevic · Paperfoot

Paper

A study audited how evaluation code extracts yes-or-no answers to measure hallucination in vision-language models on the POPE benchmark, testing the LLaVA-1.5-7B model across 9,000 questions under 4-bit NF4 quantization. A two-token logit readout checked two vocabulary IDs for "yes" and "no" that the model never output, because SentencePiece text encoding prepended a space after the prompt suffix. Switching to an eight-token lookup covering space-prefixed words yielded F1 scores of 0.8221 on the adversarial split, 0.8498 on the popular split, and 0.8713 on the random split. That 6.13-point adversarial shift exceeded the gains reported by five of six surveyed papers. Testing nine inference-time fixes on this baseline yielded none that beat it. An audit across four further vision-language models confirmed the confound is protocol-level: copying the original token IDs dropped F1 to 0.00 on two models with disjoint vocabularies, while tokenizer-derived readouts scored between 0.80 and 0.92.

Why it matters

The study shows on a public benchmark that readout choices can shift hallucination scores by more than published fixes. Because surveyed papers did not report their readout methods, whether reported gains hold under matched evaluation remains unresolved.

Caveats

The reference baseline was evaluated under 4-bit NF4 quantization rather than full precision, leaving comparisons unresolved until readout code and precision are matched. The paper is an unreviewed preprint.

From the paper

Limitation

The mismatch we report has a narrow technical surface: one logit comparison between two IDs that LLaVA-1.5-7B does not produce after ASSISTANT: under greedy decoding.

7 Limitations and Conclusion

Quoted word for word from the full text on arXiv.

In this paper

Methods
eight-token protocol (this paper)dynamic_single (this paper)legacy_8tok (baseline)string_parse (baseline)LLaVA-1.5-7B (evaluated)LLaVA-1.6-Mistral-7B (evaluated)InstructBLIP-Vicuna-7B (evaluated)mPLUG-Owl2-LLaMA2-7B (evaluated)Qwen2-VL-7B-Instruct (evaluated)
Datasets
POPE (evaluated)

Results

LLaVA-1.5-7B, F1, on POPE, adversarial split: 0.8221; LLaVA-1.5-7B, F1, on POPE, popular split: 0.8498; LLaVA-1.5-7B, F1, on POPE, random split: 0.8713

Read with an eight-token lookup that covers the space-prefixed surface forms, LLaVA-1.5-7B scores F1 0.8221, 0.8498, and 0.8713 on the adversarial, popular, and random splits.

Abstract

LLaVA-1.5-7B, F1, on POPE, adversarial split: 82.21

The LLaVA-1.5-7B baseline we measured, 82.21 F1 on the adversarial split, sat 1.21 to 8.67 points above the adversarial baselines we planned to compare against (81.00, 79.30, and 73.54).

Abstract

What it shows

Porting fixed LLaVA-1.5 token IDs verbatim drops F1 to 0.00 on disjoint-vocabulary models, whereas a tokenizer-derived readout holds between 0.80 and 0.92 across evaluated models and splits.

A four-readout audit across four further VLMs from three tokenizer families confirms the confound is protocol-level, not model-specific: porting the LLaVA-1.5 IDs verbatim drops F1 to 0.00 on the two disjoint-vocabulary models, whereas a tokenizer-derived readout holds between 0.80 and 0.92.

Abstract

In a 3,000-question probe on the adversarial split, incorrect predictions draw more image attention than correct predictions.

In a 3,000-question probe, wrong predictions draw more image attention than correct ones (Cohen's d=-0.46, p=1.4×10−24), the opposite of what grounding-based corrections assume.

Abstract

Published comparisons cannot be resolved because papers omit their readout convention and precision levels.

The 6.13-point shift matters because it exceeds the adversarial gain reported by five of the six papers we surveyed, and none of those papers states which readout produced its baseline.

1 Introduction

String parsing matches dynamic single-token logit readout across models except when decoded text fails to begin with the answer token.

The string_parse column matches dynamic_single to four decimals on LLaVA-1.6-Mistral, mPLUG-Owl2, and Qwen2-VL.

5 Multi-Model Validation

Quoted word for word from the full text on arXiv.

The paper

Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation

Kesav Kumar Jayakumar, Karthigeyan Thilak

arXiv, Oct 19, 2026, Open access, CC BY, Preprint, not peer-reviewed

doi.org/10.48550/arxiv.2610.25964

arXiv:2610.25964v3 in cs.CV, also cs.CL

AbstractPDFHTML

Comments: NeurIPS 2026 VLM4RWD. 21 pages, 10 figures. Code: https://github.com/Kesav2k04/ugaa-research

Cite as BibTeX
@misc{jayakumar2026tokenset,
  title = {Token-Set Choice Confounds POPE: A Systematic Audit of Yes/No Extraction in VLM Hallucination Evaluation},
  author = {Jayakumar, Kesav Kumar and Thilak, Karthigeyan},
  year = {2026},
  eprint = {2610.25964},
  archivePrefix = {arXiv},
  primaryClass = {cs.CV},
  doi = {10.48550/arxiv.2610.25964},
  url = {https://arxiv.org/abs/2610.25964}
}

Related