ECCV 2026

Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs

Vishal Narnaware*, Animesh Gupta*, Kevin Zhai, Zhenyi Wang, Mubarak Shah

Institute of Artificial Intelligence, University of Central Florida, USA

*Equal contribution

Paper GitHub BibTeX

VISAGE is a training-free decoding method that re-ranks token commitments in Multimodal Diffusion Large Language Models using the spatial entropy of cross-attention.

Read abstract
Multimodal Diffusion Large Language Models (MDLLMs) achieve high-concurrency generation through parallel masked decoding, yet the architectures remain prone to multimodal hallucinations. This structural vulnerability stems from an algorithmic flaw: the decoder ranks candidate tokens based on textual likelihood without verifying localized visual support. We establish that this language-only ranking induces an objective mismatch, where language probability mass acts as a misspecified proxy for the intended multimodal task. Consequently, we reinterpret hallucination as a localized optimization error, a phenomenon where the decoder exploits language shortcuts to maximize a proxy score at the expense of visual grounding. To address this objective mismatch, we introduce VISAGE, a training-free decoding framework that calibrates the objective at inference time. VISAGE estimates the proxy discrepancy by quantifying the spatial entropy of cross-attention distributions. By enforcing a localization consensus across attention heads, the method penalizes spatially uniform distributions and re-ranks token commitments to favor visually grounded outcomes. We provide an analytical stability guarantee establishing that VISAGE maintains a bounded objective loss under estimation error. Evaluations across hallucination-sensitive and general-purpose benchmarks demonstrate the robustness of the framework, yielding relative gains of 8.59% on MMMU-val and 7.75% on HallusionBench.
Figure 1. Standard decoding prematurely commits the high-confidence token "no." VISAGE estimates the proxy discrepancy and corrects the ranking score, allowing the grounded response to be recovered.
+8.59%MMMU-valrelative gain
+7.75%HallusionBenchrelative gain
Training-freeNo fine-tuning 

Motivation

Language shortcuts in parallel decoding

Standard parallel decoding ranks candidate positions using token confidence \(c_i\). This score does not include an explicit term that verifies localized visual correspondence before commitment.

\[ \mathcal{U}_t \;=\; \arg\max_{\substack{\mathcal U\subseteq\mathcal C_t\\|\mathcal U|=k_t}} \sum_{i\in\mathcal U}\log c_i \]

We compare the normalized peak probability mass of the visual and language components before token commitment. For grounded tokens, the visual component exceeds the language component before commitment. For hallucinated tokens, the visual component remains suppressed relative to the language component.

We therefore use the spatial Shannon entropy of cross-attention as a position-wise signal for re-ranking ungrounded commitments.

Method

VISAGE: Grounding-Aware Decoding

VISAGE modifies the commitment ranking without changing the underlying MDLLM or its token proposals.

Figure 3. Overview of VISAGE during parallel masked decoding.
01 Visual attention

For each candidate position, retain its token confidence \(c_i\) and extract last-layer cross-attention over the image tokens.

02 Spatial entropy

Normalize the image-token attention independently for each head and compute its Shannon entropy. Lower entropy indicates more localized visual support.

03 Head consensus

Aggregate the head-wise entropies with a β-quantile to obtain the robust grounding entropy \(H_i\). This reduces sensitivity to isolated sharp heads.

04 Re-rank commitment

Combine \(H_i\) with the original confidence and select the top-\(k\) positions using the corrected ranking score.

\[ u_i \;=\; c_i\,(1+H_i)^{-\alpha} \]
\(c_i\)token confidence
\(H_i\)robust grounding entropy
\(\alpha\)penalty strength

Higher grounding entropy decreases the commitment score before the standard top-k selection.

Analytical stability bound → Supplementary Sec. A

Results

Evaluation across multimodal benchmarks

+8.59%MMMU-valrelative
+7.75%HallusionBenchrelative
+0.26%POPErelative
−0.81%MMErelative
Table 1. Main results on MMaDA. Best value per metric in bold.
Method MMMU-val (Acc % ↑) HallusionBench (Acc % ↑) POPE (F1 ↑) MME (Score ↑)
MMaDA (Base) 27.1134.1875.971383.29
MMaDA + VCD 28.4434.8075.851342.21
MMaDA + VISAGE 29.4436.8376.171372.05
Relative gain (Ours vs. Base) +8.59%+7.75%+0.26%−0.81%

HallusionBench categories

Figure 4. VISAGE improves several spatial and visually deceptive categories. The baseline remains stronger on video, where correct reasoning requires cross-frame temporal consistency.

VISAGE uses frame-level spatial grounding and does not explicitly enforce temporal consistency across frames.

Cross-model generalization

Table 2. Best value per backbone and metric in bold.
BackboneMethodMMMU-val ↑MME ↑
LLaDA-VBase46.671883.57
LLaDA-V+ VISAGE47.11+0.441900.64+17.07
Lumina-DiMOOBase29.331125.23
Lumina-DiMOO+ VISAGE30.13+0.801142.16+16.93

VISAGE improves LLaDA-V and Lumina-DiMOO without architecture-specific tuning.

Qualitative results

Qualitative comparison with standard decoding

Visual relation1 / 13

Question

Is the spoon in the bowl? Please answer yes or no.

Ground truth

yes

MMaDA

<think> The image shows a bowl of broccoli with a lemon slice and a lemon wedge in the background. The spoon is not inside the bowl, as it is placed outside the bowl.</think> No

MMaDA + VISAGE

<think> The image shows a bowl of broccoli with a lemon slice and a lemon wedge in the background. The spoon is placed in the bowl, indicating that it is in the bowl.</think> Yes

Prompt-driven language shortcut2 / 13

Question

Are there any carrots in this figure?

Ground truth

No, all the items here are engineering tools

MMaDA

<think> The image shows a row of kitchen tools, including spatulas, spatulas, and a tape measure. There are no carrots visible in the image. The answer is no.</think> No

MMaDA + VISAGE

<think> The image shows a row of tools, including pliers, wrenches, and other hand tools, arranged on a white background. There are no carrots visible in the image. The tools appear to be related to construction or repair work.</think> No

Both methods produce the correct final answer; the baseline hallucinates irrelevant objects in its intermediate reasoning.

HallusionBench · perspective3 / 13

Question

Are these two men in the images the same height? Yes or No

Ground truth

Yes, these two men are the same height.

MMaDA

<think> To determine if the two men in the images are the same height ... From the image, we can see that the man on the left is standing with his hands in his pockets, and the man on the right is standing with his arms at his sides ... Therefore, the two men are not the same height.</think> No

MMaDA + VISAGE

<think> To determine if the two men in the images are the same height, we need to compare the silhouettes of the men ... This suggests that the men are the same height. Therefore, the answer is: Yes </think> Yes

HallusionBench · color illusion4 / 13

Question

The image shows a color illusion. Are two circles in the image different color? yes or no

Ground truth

No, two circles are the same color.

MMaDA

<think> To determine if two circles in the image are different colors, we need to analyze the color illusion provided. The image shows two circles with different colors: blue and yellow ... and the colors of the circles are distinct. Therefore, the answer is: yes </think> yes

MMaDA + VISAGE

... The image shows a consistent pattern of blue and yellow circles ... occurs when the colors are arranged in such a way that they appear to be the same color. Therefore, two circles in the image are the same color. </think> no

HallusionBench · geometry5 / 13

Question

Whether lines of blocks are parallel? Yes or No

Ground truth

No, lines of blocks are not parallel

MMaDA

<think>To determine whether lines of blocks are parallel, we need to consider the structure of a checkerboard pattern... In this pattern, each row is represented by a block, and each row is parallel to the previous row. Therefore, the lines of blocks are parallel. </think> yes

MMaDA + VISAGE

<think> To determine whether lines of blocks are parallel, we need to analyze the arrangement... The blocks are arranged in a grid-like pattern... This means that the lines of blocks are not parallel. Therefore, the answer is: No </think> no

MMMU-val · visual motif6 / 13

Question

Refer to the figure <image 1>, which term refers to adventure stories with a search motif?

A. FableB. Humorous TaleC. ThemeD. Quest Stories

Ground truth

D

MMaDA

<think> The term that refers to the adventure stories with a search motif is "Fable." The image shows a collection of adventure stories, so the correct answer is Fable. </think> A

MMaDA + VISAGE

<think> The term that refers to the adventure stories with a search motif is "quest Stories." </think> D

MMMU-val · reading comprehension7 / 13

Question

Which of the following options best relates to the poem 'Dreams' <image 1>?

A. The father is compared to a gypsy.B. The danger of letting go of a dream.C. Keep hold of your dream.D. Let's celebrate bold women!

Ground truth

C

MMaDA

<think> To determine which option best relates to the poem 'Dreams', let's analyze the content of each option... B. The danger of letting go of a dream: This option is relevant to the poem, as it directly addresses the poem's central theme.... Based on this analysis, the correct answer is B.</think> B

MMaDA + VISAGE

<think> To determine which option best relates to the poem 'Dreams', let's analyze the content of each option... C. Keep hold of your dream: Holding your dream tight is a direct interpretation of the poem's message, which is to hold onto and not let go of your dreams. Based on this analysis, the correct answer is: C

MMMU-val · historical reasoning8 / 13

Question

The <image 1> represents which of the following characteristics of the period 600-1450 C.E.?

A. Japanese women exercised influenceB. Japanese women had equality.C. Japanese society was patriarchal.D. Chinese views influenced Japan.

Ground truth

A

MMaDA

<think> To determine which characteristic represents the period 600-1450 C.E., we need to analyze the image: 1. The image shows a statue of a woman, a well-known figure in Japan, sitting on a bench ... the correct answer is: C. Japanese society was highly patriarchal, much like Chinese society. Therefore, the answer is: C </think>C

MMaDA + VISAGE

<think> To determine which characteristic represents the period 600-1450 C.E., let's analyze each option: Option A: "Japanese women exercised influence on society"- This statement is a significant fact in Japan during the period, Women had a significant role in society, despite the influence of men in other areas...Therefore, the correct answer is: A

POPE · annotation noise9 / 13

Question

Is there a toothbrush in the image?

Dataset annotation

yes

MMaDA

<think> The image shows a bathroom with a toilet, a sink, and a mirror above the sink. The sink has a faucet and a soap dispenser. There is a black cabinet with a drawer below it. The wall is painted white, and there is a light switch on the wall. The door is open, revealing a hallway. </think> yes, there is a toothbrush in the image.

MMaDA + VISAGE

<think> The image shows a bathroom with a toilet, a sink, and a mirror above the sink. The sink has a faucet and a soap dispenser. There is a black cabinet with a drawer below it. The wall is painted white, and there is a light switch on the wall. The door is open, and the floor is tiled.</think> no

The paper notes that the ground truth falsely asserts a toothbrush is present; VISAGE refutes the object and is penalized as a false negative.

POPE · label granularity10 / 13

Question

Is there a truck in the image?

Dataset annotation

yes

MMaDA

<think> The image shows a white van with a satellite dish on top, parked in front of a red brick building. The van appears to be a mobile home or a utility vehicle, possibly used for communication or broadcasting purposes ... </think> yes, there is a truck in the image.

MMaDA + VISAGE

<think> The image shows a white van with a satellite dish on top, parked in front of a red brick building. The van appears to be a mobile home or a utility vehicle, possibly used for communication or broadcasting purposes ... </think> no, there is no truck in the image.

The paper notes that the annotation loosely categorizes the white utility van as a “truck”; VISAGE identifies it as a van and rejects the generic label.

POPE · missing annotation11 / 13

Question

Is there a bottle in the image?

Dataset annotation

no

MMaDA

<think> The image shows a bathroom with a green tiled wall, a black bench, and a mirror above the sink. There is a shower area with a glass door, and the floor is tiled. The lighting is bright, and the overall setting appears to be a modern, clean bathroom. There is no visible bottle in the image.</think> no, there is no bottle in the image.

MMaDA + VISAGE

<think> The image shows a bathroom with a green tiled wall, a sink, and a mirror. There is a glass shower enclosure with a silver handle, and a small shelf with a bottle on it. The floor is tiled, and there is a wooden chair in the foreground. The lighting is bright, and the overall setting appears clean and modern.</think> yes, there is a bottle in the image.

The paper notes that the ground truth does not account for the bottle on the shower shelf, so the visually grounded VISAGE response is scored as incorrect.

Temporal reasoning · failure case12 / 13

Question

He is running counterclockwise. According to the positive sequence of the images, are they in the correct order? Answer in one sentence.

Ground truth

The images are not in correct order

MMaDA

<think> The sequence of images appears to be in the correct order, as the first image shows a crowd of people in a stadium, the second image shows a person on a track, and the third image shows a person on a track. This sequence suggests that the person is running counterclockwise, as indicated by the text provided.</think> Yes

MMaDA + VISAGE

<think> The sequence of images appears to be counterclockwise, as indicated by the movement of the crowd and the direction of the camera. The sequence of images is counterclockwise.

Both methods fail to verify the shuffled chronological order. VISAGE uses spatial attention and does not explicitly enforce cross-frame temporal consistency.

Temporal reasoning · failure case13 / 13

Question

They are hugging. According to the positive sequence of the images, are they in the correct order? Answer in one sentence.

Ground truth

The images are not in correct order

MMaDA

<think> The image shows two individuals embracing, with one person in the foreground and the other in the background ... The text overlay at the bottom of the image reads "NO." </think> NO

MMaDA + VISAGE

<think> The image shows two individuals embracing, with one person in the foreground and the other in the background ... The text overlay indicates that the image is in the correct sequence, with the person in the foreground being the one in the correct order.</think> Yes, they are in the correct order.

The baseline reaches the correct final label but hallucinates a non-existent “NO” overlay; VISAGE also fails to detect the temporal inconsistency.

View all qualitative examples in supplementary →

Analysis

Ablation studies

α

Penalty strength

αMME score ↑
0.51320.77
0.31372.05
0.11362.68

α = 0.3 gives the best MME score in this sweep.

β

Head consensus

AggregationMMMU-val ↑
Min over heads28.56
Mean over heads28.78
qβ (Ours)29.44

β-quantile aggregation outperforms minimum and mean aggregation.

s

Entropy stride

Methods/image ↓Rel. ↓
MMaDA (Base)8.221.00×
Every step10.841.31×
Stride 29.561.16×
Stride 48.581.04×
Stride 88.231.00×

Stride 8 reduces latency to 8.23 s/image, approximately baseline cost.

Citation

BibTeX

BibTeX 
@article{narnaware2026seeing,
  title={Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMs},
  author={Narnaware, Vishal and Gupta, Animesh and Zhai, Kevin and Wang, Zhenyi and Shah, Mubarak},
  journal={arXiv preprint arXiv:2603.25711},
  year={2026}
}