Learning to Predict Middle-Layer Attention in MLLMs for Visual Token Pruning

Abstract

Multimodal large language models (MLLMs) achieve strong performance across diverse vision-language tasks, but their efficiency is limited by the cost of processing numerous visual tokens. Visual token pruning can reduce this cost, but requires accurate token importance estimates. Recent studies have demonstrated that text-to-vision attention from middle language model layers can effectively guide visual token pruning, typically using attention from a predefined middle layer to select the visual tokens to retain. Two problems therefore remain. First, our analysis shows that the layer whose attention is most responsive to the question varies substantially across samples, making a fixed layer suboptimal. Second, obtaining attention from the appropriate middle layer requires processing numerous visual tokens through several language model layers, by which point considerable computation has already been spent. To address both problems, we propose Middle-layer Attention Prediction (MAP), which uses Question Contrastive Teacher Selection to identify a sample-specific teacher layer by contrasting attention under the original and reference questions, and distills attention from the selected layer into a lightweight predictor that estimates visual token importance from multi-modal input features. During inference, MAP combines the predicted importance scores with a diversity criterion to prune visual tokens before the first language model layer. Thus, MAP requires no attention maps for pruning and remains compatible with existing inference acceleration techniques. Across ten benchmarks on LLaVA-NeXT-7B, MAP retains 97.5% of the unpruned model performance with only 5.56% of the visual tokens, yielding a 3.09x end-to-end speedup.

Authors 6

Yuyao Sun, Tao Deng, Shuang Li, Deqing Wang, Hao Geng, Minjun Yu

Specification

Official page

arxiv.org/abs/2608.06411

Source:arXiv (Atom API + RSS)·T1·observed 2 h ago·high

Arxiv announce type

replace

Source:arXiv (Atom API + RSS)·T1·observed 2 h ago·high

arXiv id

2608.06411

Source:arXiv (Atom API + RSS)·T1·observed 2 h ago·high

Categories

cs.AI, cs.CV

Source:arXiv (Atom API + RSS)·T1·observed 2 h ago·high

PDF

arxiv.org/pdf/2608.06411

Source:arXiv (Atom API + RSS)·T1·observed 2 h ago·high

Primary category

cs.AI

Source:arXiv (Atom API + RSS)·T1·observed 2 h ago·high

Published

11 Sept 2026

Source:arXiv (Atom API + RSS)·T1·observed 2 h ago·high

Each value shows its source, tier and observation time. Conflicting claims are kept side by side and flagged — never averaged. How AI Atlas records facts →

Provenance