A simple style change can nudge a vision-language model toward a different conclusion, according to new research from five institutions in Japan, Germany, and the UK.
The paper, Seeing Red, Thinking Bad: Color Bias in Vision Language Models, comes from Japan’s AIST, the University of Tsukuba, the University of Technology Nuremberg, and the University of Oxford. The team built Stealth Visual Prompts, ordinary-looking formatting changes that carry no explicit instruction, then tested four open-source models.
Qwen2-VL-7B proved most susceptible. Coloring positive words green or blue pushed sentiment up to +0.42, while red negative words dragged it down to -0.48, a total swing of 0.90 versus an all-black baseline. IDEFICS2-8B showed a range of 0.52, while both LLaVA variants barely moved at 0.04 to 0.12.
A CLIP analysis showed why: changing a word’s hue shifted its internal semantic representation, most strongly along the good-versus-bad axis, even though the text never changed.
Contrast works too. In question-answering tests, making an incorrect decoy answer visually prominent reduced accuracy as surrounding text faded, and induced error rates grew as readability dropped.
The authors argue the vulnerability matters for any pipeline that feeds documents or UI screenshots into models, and they suggest defenses: normalize rendered text before inference, cross-check image-based answers against OCR output, and add style-invariance checks to evaluation suites.
