# ReToken：一个token提升视觉语言模型的视觉检索能力

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-07-30 08:00
- AIHOT 分数：68
- AIHOT 链接：https://aihot.virxact.com/items/cms8ckrga0822rot0b9akufhn
- 原文链接：https://arxiv.org/abs/2607.28627

## AI 摘要

ReToken通过一个可学习的嵌入token作为检索目标，从预填充的视觉KV缓存中筛选与查询相关的稀疏视觉token。在Visual Haystacks基准上，ReToken使Qwen3VL-8B提升13.4分、InternVL3.5提升12.4分；在LVBench上零样本迁移至长视频任务，为Qwen3VL-8B带来8.0分提升。

## 正文

This work was done while the author was at UIUC (currently at Google DeepMind).

University of Illinois at Urbana-Champaign,

{yaox11, dhoiem}@illinois.edu

Abstract

Long visual context poses a challenge for vision-language models: performance degrades as the number of distractors grows, and processing all tokens at once is computationally infeasible under GPU memory constraints. We present ReToken, a single learnable embedding trained as an explicit retrieval target that selects a sparse set of query-relevant visual tokens from a pre-filled visual KV cache. Trained on only a small image-QA dataset, ReToken yields consistent gains across image and video benchmarks: on Visual Haystacks it improves Qwen3VL-8B by 13.4 points and InternVL3.5 by 12.4 points (>20% relative), and on LVBench it transfers zero-shot to long video for an 8.0-point gain with Qwen3VL-8B. Thanks to its lightweight design, both training and long-video inference fit on a single H100. Code is available at: https://github.com/avaxiao/ReToken.

(a) Attention score fails to distinguish relevant frames from distractors.

(b) ReToken achieves more than 3x recall in the last layer. Layer-wise VLM Recall@1 on QAEgo4D

Figure 1: Attention-based retrieval mechanisms are not effective for retrieving relevant visual information in VLMs. (a) shows attention scores from question tokens (as queries) to frame tokens (as keys), across frames in one video; (b) shows whether the top-1 retrieved frame falls within the ground-truth time window (240 candidate frames per video on average, sampled at 0.5 FPS).

1 Introduction

Long visual contexts, e.g. from image collections or hour-long videos, are now within the input range of vision-language models (VLMs) Liu et al. (2023); Team et al. (2024b); Bai et al. (2025); Chen et al. (2024b). However, VLMs have difficulty answering questions from long visual contexts when only a small subset of images or frames is relevant to the prompt Wu et al. (2024), and sometimes processing the full context at once is computationally infeasible under GPU memory constraints. So, handling long visual input reduces to a retrieval problem: selecting a small subset of frames or tokens from which the model can produce a correct answer. When processing text, the attention signals produced by large language models (LLMs) can be selective with respect to the input context, thanks to extensive long-context training Xiao et al. (2023); Han et al. (2024). For VLMs, we find that the analogous signal is unreliable: attention between text and visual features is weakly correlated with relevance, as shown in Fig. 1(a), and the attention-based retriever achieves only 5.1% average recall@1 across layers on QAEgo4D for Qwen3VL-8B (Fig. 1(b)).

Looking at what does work, we find a striking asymmetry: matching a precise target phrase against the average visual value projections, rather than against the keys, increases recall@1 from 65.7 to 78.0 on Qwen3VL and from 78.8 to 83.8 on InternVL3.5 in a controlled two-image setting (Tab. 1). Values carry the content that is actually propagated through attention, and they appear to provide a better space for text-based visual retrieval. However, value–value pooling does not universally outperform query–key scoring for arbitrary retrieval text. Simply averaging over all question tokens introduces noise, reversing the advantage of the value space.

Building on this finding, we propose ReToken, a single learnable embedding that is appended to the question and trained explicitly as a retrieval target. ReToken scores each frame by the cosine similarity between its projected embedding and the frame’s mean value vector at the final layer, and is supervised with a class-balanced binary cross-entropy loss against ground-truth relevance labels. The token and a single projection matrix are the only added parameters; the VLM is frozen by default.

Despite its minimal footprint, ReToken yields consistent gains across image and video benchmarks. On Visual Haystacks Wu et al. (2024), it improves Qwen3VL-8B Bai et al. (2025) by 13.4 points and InternVL3.5 Wang et al. (2025b) by 12.4 points, corresponding to over 20% relative gain. More notably, although trained only on multi-image QA, ReToken generalizes to improve performance on long-video understanding: it transfers zero-shot to long video, yielding an 8.0-point improvement on LVBench Wang et al. (2025a) with Qwen3VL-8B, where the average video length exceeds an hour.

•

We identify that retrieval scores computed in the value space, rather than the conventional query-key space, provide a substantially stronger signal for retrieving visual information.

•

We introduce ReToken, a lightweight learnable retrieval token that enables pretrained VLMs to identify relevant visual information more effectively.

•

We show that ReToken trained only on multi-image QA transfers zero-shot to long-video benchmarks, suggesting a practical path toward scalable long-context multimodal reasoning.

2 Related Work

Visual Retrieval. The dominant approach to visual retrieval is to train image-text embedding models that align visual and language feature spaces, either with dual encoders such as CLIP Radford et al. (2021), Perception Encoder Bolya et al. (2025), and SigLIP2 Tschannen et al. (2025), or by adapting a VLM into a universal embedder, as in E5-V Jiang et al. (2024) and LamRA Liu et al. (2025). While effective, all of these operate as external retrievers separate from the VLM: relevant images must first be selected by the retriever and then re-encoded by the VLM for answer generation. A related family in long-video QA chains a separate localizer with an answerer model: SeViLA Yu et al. (2023) repurposes BLIP-2 Li et al. (2023) as both a keyframe localizer and an answerer, while VideoAgent Wang et al. (2024b) and VideoTree Wang et al. (2025c) build agentic pipelines that iteratively retrieve and caption keyframes for an LLM. A second line of work, originating in long-context language modeling, instead treats the model’s own attention scores as a retrieval signal, unifying retrieval and generation within a single forward pass. InfLLM Xiao et al. (2024) and EM-LLM Fountas et al. (2025) demonstrate this effectively in NLP by selecting key-value blocks based on query-to-key attention, and ReKV Di et al. (2025) transfers the mechanism to the visual domain without modification. We diagnose the limitations of attention-based retrieval in VLMs and propose ReToken, which performs retrieval in the value space via a learnable token.

Long Context Understanding. When handling long visual contexts, prior work follows three main directions. Memory-based methods such as MovieChat Song et al. (2024) and MA-LMM He et al. (2024) maintain a fixed-size memory bank over streaming input, merging or evicting older content to bound state. Token compression methods such as Chat-UniVi Jin et al. (2024), LLaMA-VID Li et al. (2024), LongVU Shen et al. (2024), and Video-XL Shu et al. (2025) prune, merge, or summarize visual tokens before the LLM to shorten the prefix and reduce attention and KV-cache cost. Both families bound the visual context independently of the query. Retrieval-based methods instead defer context selection to inference time, picking a query-relevant subset to preserve fine-grained evidence: Goldfish Ataallah et al. (2024) chunks the video into clips and retrieves the top- by caption-query similarity; Video-RAG Luo et al. (2024) augments retrieval with visually-aligned auxiliary text from ASR, OCR, and object detectors; and DrVideo Ma et al. (2025) converts the video into a long document and retrieves question-relevant passages for the LLM. End-task accuracy in this family, however, is bottlenecked by the retriever rather than the LLM, and our work targets this bottleneck directly.

Learnable Tokens for Visual Aggregation. ReToken is most directly related to methods that introduce learnable tokens into a VLM to aggregate or retrieve visual content. The Q-Former in BLIP-2 Li et al. (2023) and the Perceiver Resampler in Flamingo Alayrac et al. (2022) use a small set of learnable queries to compress variable-length visual features into a fixed-size representation for a frozen LLM, and InstructBLIP Dai et al. (2023) extends Q-Former to be text-conditioned. These modules are trained jointly with vision-language alignment as part of the bridge between encoder and LLM, and produce many tokens (typically 32 or 64) intended to carry the visual content forward into generation. SPRING Zhu et al. (2025) prepends pluggable soft-prompt tokens to externally retrieved passages to help a frozen text-only LLM consume them. The visual summarization token in Video-XL Shu et al. (2025) similarly compresses a chunk of visual KVs into a single token. ReToken differs along three axes: (i) operational mechanism. Q-Formers and prompt tuning use learnable tokens as static, query-agnostic input conditions. In contrast, the ReToken is generated dynamically. A first-pass placeholder produces an output token that retrieves visual KV contexts relevant to the query. (ii) supervision. Q-Formers are trained as visual compressors under alignment or generation losses, while ReToken is supervised by an explicit retrieval loss scored directly against final-layer value vectors. (iii) capacity. Our method needs only a single token rather than a set of 32–64. The second half of the contribution is diagnostic: attention (query–key) scores are unreliable for visual retrieval in pretrained VLMs, while the value space carries a much stronger signal.

3 Approach

In this section, we begin by analyzing the limitations of attention-based retrieval methods for identifying relevant visual tokens in Sec. 3.1. Motivated by these insights, we then discuss our proposed ReToken approach in Sec. 3.2, which uses a learnable token embedding to help compute more informative retrieval scores over relevant visual tokens.

3.1 Attention-Based Retrieval

In our empirical setting, we consider a VLM based on a decoder-only LM comprising transformer layers . Given input question tokens and visual tokens , the VLM autoregressively generates the response based on the conditional probability . For multi-image and video inputs, the visual tokens are partitioned into frames, , with each frame contributing tokens (where the total number of visual tokens). For long videos, can be prohibitively large, rendering full-context inference infeasible. Thus, our goal is to retrieve a subset of the most relevant frames, where , whose tokens suffice to answer the question.

Attention Scores as a Retrieval Signal. At each layer , the VLM computes attention scores , which indicate how much each token contributes to the final response. Tokens with higher attention contributions can be interpreted as being more relevant to the question. For our analysis, we use the state-of-the-art ReKV Di et al. (2025) approach that directly uses attention score as the retrieval score and aggregates it at the frame level. At the th layer, we compute the mean key vector of each frame’s visual tokens and the mean query vector over the question tokens as:

(1)

where denotes the number of total question tokens. Then, the frame-level retrieval score and top- frame selection are computed as:

(2)

where denotes the frames most relevant to the question at layer . Given the per-layer retrieval sets , the model re-runs generation through the frozen VLM with layer-wise sparse attention: at each layer , the question and answer tokens attend only to the visual tokens belonging to . The reduced visual context at layer is therefore , and the answer is decoded autoregressively over this layer-dependent context .

Limitations of Attention-Based Retrieval. While attention-based retrieval is intuitive, two structural issues motivate our approach. First, attention is trained for next-token prediction rather than retrieval, so high-attention tokens are not guaranteed to correspond to query-relevant content. This mismatch is exacerbated by the composition of typical VLM training data: the input images or videos are almost always fully relevant to the question, so the model is never required to select among visual inputs. The consequences are visible in both image and video settings: on Visual Haystacks Wu et al. (2024) the layer-wise retriever lands at recall@1 even in the simplest two-image case (Tab. 1), and on long-video QAEgo4D it averages only recall@1 across layers (Fig. 1(b)). Second, averaging the question tokens to form a single query is itself a heuristic, and the resulting rankings shift with the choice of retrieval text. As shown in Tab. 1, replacing the full question sentence with a target phrase that directly names the entity of interest improves Recall@1. Fig. 2 illustrates a question sentence and its target phrase.

Figure 2: Retrieval Example. Question/Sentence: "For the image with a cow, is there a truck?". Target phrase: "a cow".

媒体内容 · 前往原文查看

Table 1: is more informative, but it is sensitive to the input. Recall@1 with 2 input images, retrieving 1 image based on the retrieval score.

Retrieval Text Qwen3VL InternVL3.5

Sentence ✓ 63.3 78.5

✓ 62.6 75.6

Target Phrase ✓ 65.7 78.8

✓ 78.0 83.8

Why Values Carry Retrieval Signal. A more reliable signal sits in the value projections. Within a transformer attention layer, the value of a token carries the content propagated to any token that attends to it, while the query-key inner product only determines how that content is aggregated. Pooling the value projections within each frame therefore yields a representation of the content the frame contributes to attending tokens, making value features more sensitive to the retrieval text than key features. Tab. 1 confirms this consistently across both backbones: with a precise target phrase that names the entity of interest, value-space pooling identifies the ground-truth image at recall@1 vs. for the corresponding query-key score on Qwen3VL, and at vs. on InternVL3.5. This mirrors observations in visual segmentation, where value features are reported to be more informative while query-key features can be replaced by alternative aggregation signals Zhou et al. (2022); Wang et al. (2024a); Lan et al. (2024); TextRegion Xiao et al. (2025) in particular shows that, in image-text models, value features in the final attention block are rich in visual-language semantics, whereas attention weights primarily serve as an aggregation mechanism. This sensitivity cuts both ways, however: with the full question sentence, averaging mixes in many uninformative tokens, and the resulting noisy query erases or even reverses the value-space advantage. The value space thus rewards a precise retrieval target and penalizes a noisy one.

3.2 ReToken: One Token for Visual Retrieval

To improve retrieval, we introduce ReToken, a single learnable embedding that is appended to the question and trained explicitly as a retrieval target. ReToken addresses the rephrasing instability by replacing the question-averaged query with a token learned from data, and strengthens the retrieval score by computing in the value space rather than the query–key space.

Retrieval Score. Given an input sequence with visual tokens , question tokens , and the appended retrieval token , we compute retrieval scores at the final layer via a lightweight projection , which together with constitutes the only added parameters. The retrieval score for the -th frame is the cosine similarity between the projected embedding and the frame’s mean value vector ,

(3)

For inference, we compute once at the final layer and broadcast it to all layers, rather than maintaining per-layer subsets . We do not apply this at training time, since our training data is short-context and retrieval is unnecessary.

Figure 3: Training Pipeline. The visual tokens, question tokens, and a learnable embedding (ReToken) are fed into the LLM decoder. The ReToken output from the final layer serves as a retrieval query, scoring each frame based on its cosine similarity to the frame’s pooled value features. The resulting scores are supervised using ground-truth frame-relevance labels and a class-balanced binary cross-entropy loss. By default, we train ReToken while keeping the VLM frozen. The generation loss is used only during VLM partial fine-tuning setting.

Training. We train ReToken with the VLM kept frozen by default; only the retrieval token and a single final-layer projection are updated. We supervise these parameters with a retrieval loss that compares the final-layer retrieval scores against ground-truth relevance labels .

Let and denote the sets of relevant and irrelevant images, respectively. To prevent the loss from being dominated by the irrelevant images, we apply a class-balanced binary cross-entropy in which the positive and negative terms are averaged separately,

(4)

where is the retrieval score for image at the final layer , is the sigmoid, and is a learnable logit scale parameter.

We additionally explore a partial-tuning variant in which the early layers of the VLM are tuned to give the visual representation more flexibility. In this setting, training is driven by two complementary losses, illustrated in Fig. 3. In addition to the retrieval loss above, we use a generation loss , the standard next-token prediction loss on the answer, which preserves the model’s question-answering ability and prevents the unfrozen layers from drifting under alone. The total loss is

(5)

Figure 4: Inference Pipeline. Our method first caches features for all input frames. Given a text query, ReToken retrieves the relevant frames, and the answer is generated from their cached features.

Inference. At test time, we use a two-pass retrieve-then-answer pipeline (Fig. 4). The video is first encoded once, and each question then retrieves a subset of visual KV cache for answer generation.

Cache video once: At video ingestion time, the VLM processes the visual tokens and stores the resulting per-layer KV cache in a persistent cache. For short videos, this is a single full-context forward pass. For long videos, we follow ReKV Di et al. (2025) and encode the video chunk by chunk with a sliding-window attention mask: each chunk attends only to the most recent visual tokens, while its newly produced KV states are appended to the cache. Older KV states can be offloaded to CPU memory when GPU memory is limited.

Retrieve then answer: Given a question, we run two forward passes through the frozen VLM over the cached visual context. The first pass computes the retrieval set from the final layer; the second pass generates the answer conditioned on the selected frames. We use two passes because ReToken is supervised only at (Eq. 4). The first pass enables the retrieval token to fully integrate textual and visual content, while the second pass enables attention to a consistent set of visual tokens, matching how it was trained.

First pass – retrieval. We append the retrieval token to the question and run the first pass over the cached visual KV. For short videos attends to all cached visual tokens at layer , and the index is computed at via Eq. 3. For long videos the early-layer attention budget cannot accommodate the full cache, so at each layer we restrict ’s attention as follows: (1) project the contextualized through that layer’s frozen value projection; (2) score every frame by the cosine similarity between this projection and the frame’s mean value vector at layer ; (3) restrict ’s attention to the top- frames under this score before computing layer ’s output. We use by default, which keeps each early-layer attention well within GPU memory while leaving headroom for the final-layer ranking to refine the selection. At we then compute Eq. 3 over the per-frame value means to obtain , reordered by original timestamp.

We note an asymmetry between this retrieval-pass budget and the answer-stage budget : ReToken’s retrieval pass attends to up to frames per early layer, which is different from the used at answer time. ReToken therefore has access to more visual context when ranking candidates, while it still uses only -frame visual context when generating the answer.

Second pass – answer. For each layer , we load only the visual KV cache belonging to frames in , and run a standard generation pass over this reduced visual context together with the question. The answer stage therefore attends to about visual tokens instead of all , while the expensive video encoding is performed only once per video. This is particularly efficient when multiple questions are asked about the same video. Full prompt templates and a worked example of how ReToken interacts with the contextual visual tokens and the question are provided in Supp. A.

4 Experiments

4.1 Settings

Implementation Details. We use the greedy decoding configuration for ReToken and every baseline. We train ReToken using Qwen3VL-8B and InternVL3.5-8B on an image question-answering (QA) dataset. For the default frozen-VLM setting, we use an effective batch size of 64 and a learning rate of to train the learnable token for 3 epochs on a single H100 GPU with Qwen3VL-8B. Since InternVL3.5-8B requires more tokens to represent an image, resulting in substantially higher computational and memory costs, we train InternVL3.5-8B for only one epoch. We adopt a linear warmup schedule followed by cosine learning rate decay. For the partial fine-tuning setting, we use a learning rate of and an effective batch size of 64. As this setting converges rapidly and begins to overfit after one epoch, we apply early stopping at the end of the first epoch. Training Qwen3VL-8B under this setting takes approximately 4 hours on a single H100 GPU.

For long video inference, we use an encoding chunk size of 128 frames and a sliding-window length of tokens, which corresponds to roughly 153 frames at 196 tokens per frame.

Training Datasets. Our default multi-image training dataset comes from the MIRAGE Wu et al. (2024) fine-tuning dataset, whose examples are each annotated with a relevant/irrelevant label per image. We use a 95%/5% train/validation split. It is a multi-image QA (MIQA) dataset that combines existing MIQA datasets (RetVQA Penamakuri et al. (2023), SlideVQA Tanaka et al. (2023), and WebQA Chang et al. (2022)) with synthetic MIQA data adapted from the LLaVA Visual Instruct 150K dataset Liu et al. (2024) via clustering and distractor sampling. Please refer to Supp. B.1 for more details.

Evaluation Datasets. We evaluate ReToken on four benchmarks chosen to probe complementary aspects of retrieval and long-context understanding.

Visual Haystacks (VHs) Wu et al. (2024) is a benchmark introduced alongside MIRAGE to evaluate the pure visual recognition ability of vision-language models. Constructed from the COCO dataset Lin et al. (2014), it consists of 1,000 question-answer pairs and provides query-relevant image annotations. The answer is always either “Yes” or “No”. For each QA example, a varying number of distractor images is added to test the model’s ability to locate informative content within a large pool of inputs.

QAEgo4D Di and Xie (2024) is the multiple-choice subset of the QAEgo4D-test benchmark Bärmann and Waibel (2022), focusing on question answering over long egocentric videos. Each example is annotated with the video segments relevant to the question. Video lengths range from 4 to 20 minutes.

LVBench Wang et al. (2025a) is a benchmark for extreme long-video understanding, comprising 103 publicly sourced YouTube videos totaling roughly 117 hours, with an average length of 68 minutes and individual videos extending up to 2 hours. This is a multiple-choice dataset, and the metric is accuracy.

Video-MME Fu et al. (2025) consists of 900 videos and 2,700 question-answer pairs. Video durations range from 11 seconds to 1 hour, partitioned into short (2 min), medium (4–15 min), and long (30–60 min) splits. Each question is multiple-choice, and we report accuracy as the evaluation metric.

Figure 5: Accuracy vs. Retrieve Images. Frozen Qwen3VL-8B.

媒体内容 · 前往原文查看

Table 2: Retrieval Strategy Comparison. We compare ReToken against different retrievers on Visual Haystacks with retrieval budget . “GT Cache” means using the KV cache of the ground truth image as input and serves as the oracle upper bound for any retriever. All results are based on Qwen3VL-8B with the VLM frozen.

Metric Standard GT Cache SigLIP2 ReKV CoT ReToken

Recall N/A 100 76.4 63.3 65.3 88.5

Accuracy 82.0 86.5 82.8 73.0 77.8 85.9

Recall N/A 100 20.8 1.8 3.1 64.7

Accuracy 58.6 80.7 60.7 51.2 51.5 72.0

4.2 Visual Haystacks Retrieval and Accuracy

We evaluate ReToken against the baselines on the single-needle task of the Visual Haystacks benchmark Wu et al. (2024), in which exactly one image in the haystack is relevant to the query. The context size controls the number of distractor images, with larger posing a greater challenge to the model. denotes the number of retrieved images.

Retrieval Behavior of ReToken and Attention-Based Retrieval. Fig. 5 compares ReToken and attention-based retriever, ReKV Di et al. (2025), across retrieval budgets from to at and . When , no retrieval is needed since all images are used as input. The two methods exhibit opposite trends in the meaningful retrieval regime (): ReToken performs best at small and degrades as more images (mostly distractors) are retrieved, whereas ReKV starts low and improves with larger . The crossover reflects a precision–recall tradeoff: ReToken is precise per slot, while ReKV needs a wider budget to recall the relevant image.

Retrieval Strategy Comparison. We also compute recall to measure the retrieval ability of different strategies on Visual Haystacks. Here, recall measures whether the retriever successfully selects the ground-truth image when . Tab. 2 compares ReToken against several retrieval strategies on Visual Haystacks. We compare against four baselines: Standard pre-fills all images (the default VLM inference pipeline); GT Cache uses the ground-truth image’s KV cache as input, serving as an oracle upper bound; SigLIP2 Tschannen et al. (2025) first retrieves a single image with siglip2-giant-opt-patch16-384 and feeds its KV cache to the VLM; and CoT prompts the VLM to first generate a target search phrase, then uses that phrase as the query for ReKV attention-based retrieval (full prompt in Supp. B.2).

ReToken substantially outperforms all baselines, retaining most of the GT Cache upper-bound accuracy at large context size () while the others fail.

媒体内容 · 前往原文查看

Table 3: Partial tuning on the retrieval dataset yields a cleaner KV cache. measures the degree of distraction. A smaller gap indicates a cleaner stored KV cache.

Model Loss GT Image GT Cache

Generation Retrieval

Qwen3VL-8B 87.8 86.5 80.7 1.3 7.1

Tuned Layer 1 - 3 ✓ 87.8 86.8 82.3 1.0 5.5

✓ ✓ 88.5 87.3 83.6 1.2 4.9

How the Stored KV Cache Differs From Re-encoding the Visual Input. Observing the accuracy for GT Cache in Tab. 2, we notice an interesting phenomenon: the accuracy degrades with higher C, even though only the cache for the same single ground truth relevant image is used in each case (86.5 for vs. 80.7 for ). This is because the image tokens have attended to previous images, which are irrelevant in this case, picking up distracting information.

To quantify this distraction, we compare the accuracy when using “GT Image” (re-encoding the ground-truth image alone, yielding a clean KV cache) with “GT Cache” (the fused KV cache for the ground-truth image). Their gap, , measures the degree of distraction (Tab. 3); a smaller gap indicates a cleaner stored KV cache. Partial tuning on the retrieval dataset encourages the model to produce cleaner KV cache representations (KV cache is not affected when we only train the retrieval token). A cleaner KV cache benefits Visual Haystacks, but may pose a challenge for video understanding, which requires connecting information across adjacent frames.

媒体内容 · 前往原文查看

Table 4: ReToken’s advantage grows with context size, yielding over 20% relative gain at C=50, when freezing the VLM. Performance on Visual Haystacks. denotes the number of context images (i.e., images provided as input). indicates a context overflow or CUDA out-of-memory error.

Method

Generalist

Gemini-1.5 Pro Team et al. (2024a) 88.4 82.0 78.3 76.0 71.9 68.6 62.8 57.4

InternVL2 Chen et al. (2024a) 88.1 80.5 72.3 63.9 58.8 55.2

Specialist

SigLIP2 Tschannen et al. (2025) 72.0 69.2 68.1 65.3 64.1 60.3 58.7 58.3

MIRAGE Wu et al. (2024) 83.2 77.8 76.6 72.8 70.5 66.0 63.6 62.0

REN Khosla et al. (2025) 81.2 78.6 77.4 76.0 74.0 72.1 68.3 65.5

Freezing the VLM

Qwen3VL-8B Bai et al. (2025) 87.8 82.0 77.9 74.7 69.2 64.0 58.6 55.7

+ ReToken 87.8 (+0.0) 85.9 (+3.9) 84.4 (+6.5) 82.5 (+7.8) 80.7 (+11.5) 76.8 (+12.8) 72.0 (+13.4) 67.7 (+12.0)

InternVL3.5-8B Wang et al. (2025b) 87.5 81.6 79.3 73.1 69.0 65.0 57.3 55.3

+ ReToken 87.5 (+0.0) 84.1 (+2.5) 83.7 (+4.4) 80.7 (+7.6) 76.9 (+7.9) 73.0 (+8.0) 69.7 (+12.4) 59.7 (+4.4)

Tuning the first 3 layers of the VLM

Qwen3VL-8B Bai et al. (2025) 88.5 85.6 82.4 79.6 76.5 71.2 64.2 61.5

+ ReToken 88.5 (+0.0) 86.5 (+0.9) 85.2 (+2.8) 84.3 (+4.7) 82.8 (+6.3) 79.8 (+8.6) 75.0 (+10.8) 70.6 (+9.1)

Accuracy Comparison. Tab. 4 reports accuracy across context sizes from (only the query-relevant image, no distractors) to . The retrieval budget is set to 1. At , ReToken matches the vanilla Qwen3VL-8B backbone, as we skip retrieval when .

As the context size grows, the gap between ReToken and the vanilla backbone widens substantially. The one exception is InternVL3.5-8B at , which we attribute to its 1-epoch training budget (Sec. 4.1). ReToken also compares favorably against external baselines, surpassing all baselines across all context sizes, including the dedicated multi-image RAG framework MIRAGE.

4.3 Image-to-Video Transfer Evaluation

Having validated ReToken on image data, we now evaluate its transfer to the video setting. Notably, results in this section are obtained with ReToken trained only on the multi-image MIRAGE training data, demonstrating strong zero-shot transfer. By default, we evaluate the frozen Qwen3VL-8B on video benchmarks with videos sampled at 0.5 FPS. “Uniformly” means loading the KV cache of uniformly sampled frames as input.

媒体内容 · 前往原文查看

Table 5: Zero-shot performance on QAEgo4D.

Uniformly ReKV ReToken

1 42.8 42.6 49.6

16 53.6 57.8 60.0

32 55.2 59.8 60.6

Accuracy on QAEgo4D. Tab. 5 shows that ReToken’s advantage is most pronounced under tight retrieval budgets. With , ReToken achieves a 6.8-point improvement over uniform sampling, while ReKV actually performs slightly worse than uniform sampling. Unlike multi-image QA where images are independent, neighboring video frames provide complementary context, so accuracy continues to climb as grows even where ReToken’s lead over uniform sampling narrows.

Long Video Understanding. The benefits of retrieval grow with video length (Tab. 5(a)). On the Short split of Video-MME, ReToken provides no gain, since these videos last only 2 minutes (60 frames at 0.5 FPS), which falls below our retrieval input budget of 100 frames and therefore does not trigger retrieval. As video length grows, however, the gain becomes substantial. ReToken achieves an 8.0-point improvement on LVBench (Tab. 5(b)).

Table 6: ReToken helps more with long videos, even when trained only with images. Freezing the VLM and only tuning ReToken.

(a) Performance gain for Qwen3VL on different split of VideoMME.

媒体内容 · 前往原文查看

(b) All results in the Process raw frames section are sourced from the technical report Bai et al. (2025).

Model # Frames LVBench Video-MME

Duration 30 – 140 min 0 – 60 min

Process raw frames

Gemini 2.5 Pro Comanici et al. (2025) 512 69.0 80.6

OpenAI GPT-5 Singh et al. (2025) 256 - 77.3

Claude Opus 4.1 Anthropic (2025) 100 - 73.3

Qwen3VL-8B Bai et al. (2025) 2 FPS 58.0 71.4

Retrieve from the stored KV cache at 0.5 FPS

Qwen3VL-8B Bai et al. (2025) 100 40.6 65.1

+ ReToken 100 48.6 (+8.0) 67.1 (+2.0)

4.4 Ablations

We ablate the design choices of ReToken with Qwen3VL-8B on Visual Haystacks by default, focusing on: (1) retrieval based on visual key or value; (2) training the token only versus partial tuning; (3) the influence of the inference setting; (4) efficiency analysis; and (5) error analysis.

媒体内容 · 前往原文查看

Table 7: Key vs. Value.

Key Value

Recall 59.4 64.7

Accuracy 70.6 72.0

We further ablate design choices on whether a single token is sufficient for retrieval in Supp. 13, and whether ReToken can skip attending to visual tokens in Supp. C.2.

Retrieval Score Based on Average Vision Key or Value. We ablate ReToken’s scoring mechanism in Tab. 7 with : “Key” trains ReToken to retrieve via the average image keys, while “Value” uses the average image values. “Value” pulls clearly ahead in both recall and accuracy. We attribute this to values carrying the content actually propagated through attention, providing a more discriminative signal for distinguishing the relevant image among many distractors. Supp. B.3 provides further evidence for this.

Train Only Token or Partial Tuning VLM Layers. Tab. 8(a) ablates the partial-tuning depth. Tuning the first few layers (1–3) gives the best accuracy while preserving high recall, indicating that early layers are where visual features can be best shaped without compromising downstream performance.

Tab. 8(b) compares the transfer ability of the frozen and partial-tuning settings. ReToken shows a clear advantage in both. However, overall performance under partial tuning is lower than under the frozen setting. A possible reason is that tuning the LLM layers on image data can cause domain shift, especially given the distribution gap between image training data and video benchmarks.

Table 8: Tuning early layers can improve performance on the image task, but it degrades performance on the video task. (a) Tuning the first few VLM layers (Layer 1–3) gives the best recall–accuracy trade-off at . (b) Training the VLM on images decreases accuracy on video; evaluated on QAEgo4D with .

媒体内容 · 前往原文查看

(a) Train on images, evaluate on images.

Tuning Layers Freezing 1 1-3 1-10 N

Recall 64.7 68.3 68.2 58.2 64.3

Acc 72.0 73.3 75.0 72.6 71.3

媒体内容 · 前往原文查看

(b) Train on images, evaluate on video.

Strategy Uniformly ReKV Di et al. (2025) ReToken

Freezing 53.6 57.8 60.0

Layer 1 - 3 50.0 55.2 58.0

媒体内容 · 前往原文查看

Table 9: Two-pass Inference.

ReKV ReToken

Single 51.2 50.4

Two 50.4 72.0

Single or Two-pass Inference. Since ReToken is trained to retrieve at the last layer, we use two-pass inference to obtain retrieval results at the final layer and then broadcast them to all layers. We ablate its effect and report the accuracy in Tab. 9 under . “Single” means each layer retrieves its own images and directly generates the response based on that. “Two” means early layers attend to all visual tokens, and the last-layer retrieval result is broadcast to generate the response. The conclusions are different for ReKV and ReToken. One possible explanation is that attention-based retrieval is suited for answering directly, so it does not rely on the final layer’s results; in contrast, ReToken is only trained with the retrieval loss at the final layer. Therefore, the default setting for ReKV is single-pass, while for ReToken it is two-pass.

媒体内容 · 前往原文查看

Table 10: Runtime vs. quality on long-video QA (QAEgo4D, 240 frames per video at 0.5 FPS, single H100). Latency is reported per question after one-time video encoding. ReToken achieves higher accuracy with only modest overhead in the retrieval pass, and identical cost for the encode and answer-stage.

Latency (second, per question) Peak GPU (GB) QA (%)

Method Encode† Retrieve / Load Answer Retrieve Answer Acc. Recall

Uniformly 16 14.68 0.081 0.169 65.1 65.4 53.6 38.0

ReToken 16 14.68 0.519 0.167 66.6 65.4 60.0 70.6

Uniformly 32 14.70 0.128 0.166 66.0 66.7 55.2 59.6

ReToken 32 14.70 0.538 0.166 67.2 66.8 60.6 81.2

†Encoding is performed once per video; answers are generated from the visual tokens of the retrieved frames.

Runtime and Memory Usage. Tab. 10 breaks down per-question cost into retrieval and answer phases, and reports video encoding separately because encoding produces a persistent KV cache shared across all questions about the same video. Encoding dominates the budget at 14.7 s per video. When few questions are asked of a video, the encoding dominates the compute time, but can be performed as a pre-process. ReToken adds roughly 0.4 seconds to the per-question retrieval and answering time, but significantly improves recall and question accuracy.

媒体内容 · 前往原文查看

Table 11: ReToken helps most when the evidence is localized and nameable, and hurts when it is dispersed across the video. Per-type accuracy on LVBench. =100. #QA denotes the number of questions per type; a question can carry multiple type labels.

Question type #QA Uniformly ReToken

Key information retrieval 291 42.3 57.7 15.4

Entity recognition 677 40.0 50.5 10.5

Reasoning 201 40.8 46.3 5.5

Temporal grounding 220 35.0 38.6 3.6

Event understanding 647 41.4 43.3 1.9

Summarization 58 36.2 31.0 5.2

Error Analysis. LVBench annotates each question with a task-type label, allowing us to characterize when ReToken works best and when it fails (Tab. 11). ReToken is strongest when the evidence is localized and nameable: key information retrieval () and entity recognition () are exactly the regimes where a precise retrieval target locks onto the relevant frames. Gains shrink where relevance depends on cross-frame or temporal structure rather than per-frame content, as in temporal grounding and event understanding. ReToken hurts on summarization, where the evidence is dispersed across the entire video; uniform coverage is the better prior for this question type.

5 Conclusion

We diagnose the limitations of attention-based retrieval in VLMs and introduce ReToken, a learnable token that improves visual retrieval. Our diagnosis points to a broader principle: in pretrained VLMs, the value space carries a stronger text-aligned signal. Despite being trained only on multi-image data, ReToken yields significant improvements on both image and long-video benchmarks, transferring zero-shot from images to videos. Both training and long-video inference fit on a single H100, making ReToken a practical step toward scalable long-context multimodal reasoning.

Limitations. ReToken requires a two-pass forward and attends to more visual context in early layers, adding slightly to the memory requirements and response time. Our training data is limited to multi-image QA. Training on video data could help ReToken better capture temporal structure and improve video understanding.

Future Work. ReToken scores each frame independently by matching the query content against per-frame value means, and relaxing this design opens several directions. First, retrieval could target sets of consecutive frames whose information emerges from their temporal combination rather than from any single frame. Second, temporally offset queries such as “what happened before I entered the room” would require a scoring mechanism aware of temporal displacement, since content matching alone tends to locate the described event rather than the frames preceding it. Third, computing the retrieval score at the token level instead of the frame level could recover evidence that occupies only a few tokens and is diluted by frame-level mean pooling.

Acknowledgments

We thank Xiaodong Liu, Sethuraman T V, and Bolin Lai for their insightful comments and helpful discussions.

This research project has benefited from the Microsoft Agentic AI Research and Innovation (AARI) grant program, and was partially supported by the Office of Naval Research under grant N00014-23-1-2383. This work also used NVIDIA GPUs at NCSA Delta through allocation CIS240059 and CIS250059 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by NSF Grants #2138259, #2138286, #2138307, #2137603, and #2138296.

References

[1] J. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y. Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynolds, et al. (2022) Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35, pp. 23716–23736. Cited by: §2.

[2] Anthropic (2025-08) System card addendum: Claude Opus 4.1. Technical report Anthropic. External Links: Link Cited by: 5(b).

[3] K. Ataallah, X. Shen, E. Abdelrahman, E. Sleiman, M. Zhuge, J. Ding, D. Zhu, J. Schmidhuber, and M. Elhoseiny (2024) Goldfish: vision-language understanding of arbitrarily long videos. In European Conference on Computer Vision, pp. 251–267. Cited by: §2.

[4] S. Bai, Y. Cai, R. Chen, K. Chen, X. Chen, Z. Cheng, L. Deng, W. Ding, C. Gao, C. Ge, et al. (2025) Qwen3-vl technical report. arXiv preprint arXiv:2511.21631. Cited by: Table 12, Table 14, §1, §1, 5(b), 5(b), 5(b), 5(b), Table 4, Table 4.

[5] L. Bärmann and A. Waibel (2022) Where did i leave my keys? — episodic-memory-based question answering on egocentric videos. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Vol. , pp. 1559–1567. External Links: Document Cited by: §4.1.

[6] D. Bolya, P. Huang, P. Sun, J. H. Cho, A. Madotto, C. Wei, T. Ma, J. Zhi, J. Rajasegaran, H. Rasheed, et al. (2025) Perception encoder: the best visual embeddings are not at the output of the network. arXiv preprint arXiv:2504.13181. Cited by: §2.

[7] Y. Chang, M. Narang, H. Suzuki, G. Cao, J. Gao, and Y. Bisk (2022) Webqa: multihop and multimodal qa. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16495–16504. Cited by: §4.1.

[8] Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, et al. (2024) Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint arXiv:2412.05271. Cited by: Table 4.

[9] Z. Chen, J. Wu, W. Wang, W. Su, G. Chen, S. Xing, M. Zhong, Q. Zhang, X. Zhu, L. Lu, et al. (2024) Internvl: scaling up vision foundation models and aligning for generic visual-linguistic tasks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24185–24198. Cited by: §1.

[10] G. Comanici, E. Bieber, M. Schaekermann, I. Pasupat, N. Sachdeva, I. Dhillon, M. Blistein, O. Ram, D. Zhang, E. Rosen, et al. (2025) Gemini 2.5: pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities. arXiv preprint arXiv:2507.06261. Cited by: 5(b).

[11] W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi (2023) Instructblip: towards general-purpose vision-language models with instruction tuning. Advances in neural information processing systems 36, pp. 49250–49267. Cited by: §2.

[12] S. Di and W. Xie (2024) Grounded question-answering in long egocentric videos. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12934–12943. Cited by: §4.1.

[13] S. Di, Z. Yu, G. Zhang, H. Li, T. Zhong, H. Cheng, B. Li, W. He, F. Shu, and H. Jiang (2025) Streaming video question-answering with in-context video kv-cache retrieval. arXiv preprint arXiv:2503.00540. Cited by: §2, §3.1, §3.2, §4.2, 8(b).

[14] Z. Fountas, M. A. Benfeghoul, A. Oomerjee, F. Christopoulou, G. Lampouras, H. Bou-Ammar, and J. Wang (2025) Human-inspired episodic memory for infinite context llms. In 13th International Conference on Learning Representations Iclr 2025, Cited by: §2.

[15] C. Fu, Y. Dai, Y. Luo, L. Li, S. Ren, R. Zhang, Z. Wang, C. Zhou, Y. Shen, M. Zhang, et al. (2025) Video-mme: the first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 24108–24118. Cited by: §4.1.

[16] C. Han, Q. Wang, H. Peng, W. Xiong, Y. Chen, H. Ji, and S. Wang (2024) Lm-infinite: zero-shot extreme length generalization for large language models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pp. 3991–4008. Cited by: §1.

[17] B. He, H. Li, Y. K. Jang, M. Jia, X. Cao, A. Shah, A. Shrivastava, and S. Lim (2024) Ma-lmm: memory-augmented large multimodal model for long-term video understanding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 13504–13514. Cited by: §2.

[18] T. Jiang, M. Song, Z. Zhang, H. Huang, W. Deng, F. Sun, Q. Zhang, D. Wang, and F. Zhuang (2024) E5-v: universal embeddings with multimodal large language models. arXiv preprint arXiv:2407.12580. Cited by: §2.

[19] P. Jin, R. Takanobu, W. Zhang, X. Cao, and L. Yuan (2024) Chat-univi: unified visual representation empowers large language models with image and video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13700–13710. Cited by: §2.

[20] S. Khosla, S. TV, B. Lee, A. Schwing, and D. Hoiem (2025) REN: fast and efficient region encodings from patch-based image encoders. arXiv preprint arXiv:2505.18153. Cited by: Table 4.

[21] M. Lan, C. Chen, Y. Ke, X. Wang, L. Feng, and W. Zhang (2024) Proxyclip: proxy attention improves clip for open-vocabulary segmentation. In European Conference on Computer Vision, pp. 70–88. Cited by: §3.1.

[22] J. Li, D. Li, S. Savarese, and S. Hoi (2023) Blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730–19742. Cited by: §2, §2.

[23] Y. Li, C. Wang, and J. Jia (2024) Llama-vid: an image is worth 2 tokens in large language models. In European Conference on Computer Vision, pp. 323–340. Cited by: §2.

[24] T. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick (2014) Microsoft coco: common objects in context. In European conference on computer vision, pp. 740–755. Cited by: §4.1.

[25] H. Liu, C. Li, Y. Li, and Y. J. Lee (2024) Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 26296–26306. Cited by: §4.1.

[26] H. Liu, C. Li, Q. Wu, and Y. J. Lee (2023) Visual instruction tuning. Advances in neural information processing systems 36, pp. 34892–34916. Cited by: §1.

[27] Y. Liu, Y. Zhang, J. Cai, X. Jiang, Y. Hu, J. Yao, Y. Wang, and W. Xie (2025) Lamra: large multimodal model as your advanced retrieval assistant. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 4015–4025. Cited by: §2.

[28] Y. Luo, X. Zheng, G. Li, S. Yin, H. Lin, C. Fu, J. Huang, J. Ji, F. Chao, J. Luo, et al. (2024) Video-rag: visually-aligned retrieval-augmented long video comprehension. arXiv preprint arXiv:2411.13093. Cited by: §2.

[29] Z. Ma, C. Gou, H. Shi, B. Sun, S. Li, H. Rezatofighi, and J. Cai (2025) Drvideo: document retrieval based long video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 18936–18946. Cited by: §2.

[30] A. S. Penamakuri, M. Gupta, M. D. Gupta, and A. Mishra (2023) Answer mining from a pool of images: towards retrieval-based visual question answering. arXiv preprint arXiv:2306.16713. Cited by: §4.1.

[31] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. (2021) Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748–8763. Cited by: §2.

[32] X. Shen, Y. Xiong, C. Zhao, L. Wu, J. Chen, C. Zhu, Z. Liu, F. Xiao, B. Varadarajan, F. Bordes, et al. (2024) Longvu: spatiotemporal adaptive compression for long video-language understanding. arXiv preprint arXiv:2410.17434. Cited by: §2.

[33] Y. Shu, Z. Liu, P. Zhang, M. Qin, J. Zhou, Z. Liang, T. Huang, and B. Zhao (2025) Video-xl: extra-long vision language model for hour-scale video understanding. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 26160–26169. Cited by: §2, §2.

[34] A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, et al. (2025) Openai gpt-5 system card. arXiv preprint arXiv:2601.03267. Cited by: 5(b).

[35] E. Song, W. Chai, G. Wang, Y. Zhang, H. Zhou, F. Wu, H. Chi, X. Guo, T. Ye, Y. Zhang, et al. (2024) Moviechat: from dense token to sparse memory for long video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18221–18232. Cited by: §2.

[36] R. Tanaka, K. Nishida, K. Nishida, T. Hasegawa, I. Saito, and K. Saito (2023) Slidevqa: a dataset for document visual question answering on multiple images. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 37, pp. 13636–13645. Cited by: §4.1.

[37] G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al. (2024) Gemini 1.5: unlocking multimodal understanding across millions of tokens of context. arXiv preprint arXiv:2403.05530. Cited by: Table 4.

[38] G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivière, M. S. Kale, J. Love, et al. (2024) Gemma: open models based on gemini research and technology. arXiv preprint arXiv:2403.08295. Cited by: §1.

[39] M. Tschannen, A. Gritsenko, X. Wang, M. F. Naeem, I. Alabdulmohsin, N. Parthasarathy, T. Evans, L. Beyer, Y. Xia, B. Mustafa, et al. (2025) Siglip 2: multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv preprint arXiv:2502.14786. Cited by: §2, §4.2, Table 4.

[40] F. Wang, J. Mei, and A. Yuille (2024) Sclip: rethinking self-attention for dense vision-language inference. In European conference on computer vision, pp. 315–332. Cited by: §3.1.

[41] W. Wang, Z. He, W. Hong, Y. Cheng, X. Zhang, J. Qi, M. Ding, X. Gu, S. Huang, B. Xu, et al. (2025) Lvbench: an extreme long video understanding benchmark. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 22958–22967. Cited by: §1, §4.1.

[42] W. Wang, Z. Gao, L. Gu, H. Pu, L. Cui, X. Wei, Z. Liu, L. Jing, S. Ye, J. Shao, et al. (2025) Internvl3. 5: advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265. Cited by: §1, Table 4.

[43] X. Wang, Y. Zhang, O. Zohar, and S. Yeung-Levy (2024) Videoagent: long-form video understanding with large language model as agent. In European Conference on Computer Vision, pp. 58–76. Cited by: §2.

[44] Z. Wang, S. Yu, E. Stengel-Eskin, J. Yoon, F. Cheng, G. Bertasius, and M. Bansal (2025) Videotree: adaptive tree-based video representation for llm reasoning on long videos. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 3272–3283. Cited by: §2.

[45] T. Wu, G. Biamby, J. Quenum, R. Gupta, J. E. Gonzalez, T. Darrell, and D. M. Chan (2024) Visual haystacks: a vision-centric needle-in-a-haystack benchmark. arXiv preprint arXiv:2407.13766. Cited by: §1, §1, §3.1, §4.1, §4.1, §4.2, Table 4.

[46] C. Xiao, P. Zhang, X. Han, G. Xiao, Y. Lin, Z. Zhang, Z. Liu, and M. Sun (2024) Infllm: training-free long-context extrapolation for llms with an efficient context memory. Advances in neural information processing systems 37, pp. 119638–119661. Cited by: §2.

[47] G. Xiao, Y. Tian, B. Chen, S. Han, and M. Lewis (2023) Efficient streaming language models with attention sinks. arXiv preprint arXiv:2309.17453. Cited by: §1.

[48] Y. Xiao, Q. Fu, H. Tao, Y. Wu, Z. Zhu, and D. Hoiem (2025) TextRegion: text-aligned region tokens from frozen image-text models. arXiv preprint arXiv:2505.23769. Cited by: §3.1.

[49] S. Yu, J. Cho, P. Yadav, and M. Bansal (2023) Self-chained image-language model for video localization and question answering. Advances in Neural Information Processing Systems 36, pp. 76749–76771. Cited by: §2.

[50] C. Zhou, C. C. Loy, and B. Dai (2022) Extract free dense labels from clip. In European conference on computer vision, pp. 696–712. Cited by: §3.1.

[51] Y. Zhu, Z. Huang, Z. Dou, and J. Wen (2025) One token can help! learning scalable and pluggable virtual tokens for retrieval-augmented large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, Vol. 39, pp. 26166–26174. Cited by: §2.

Appendix A Prompt Templates

In this section, we detail the prompt templates used during training and inference with ReToken. To highlight the modifications introduced by ReToken, we color-code the content-bearing tokens as follows: All Vision Tokens, Retrieved Vision Tokens, Question Tokens and the <Retrieval> token, and the model output Answer.

A.1 Original Prompt Template

The standard prompt format used by vision-language models feeds all visual tokens into the model alongside the textual query:

A.2 Training Prompt Template

During training, we employ two complementary prompt formats depending on the training mode. The first, which appends the <Retrieval> token after the question (teaching the model when to invoke retrieval), is used in both frozen and partially fine-tuned settings:

The second format, designed to preserve the model’s original QA abilities, is incorporated alongside the first exclusively when the LLM is partially fine-tuned:

A.3 Inference Pipeline

At inference time, we adopt a two-stage retrieve-then-answer pipeline. In Stage 1, at most frames are attended during retrieval and the model uses the <Retrieval> token to aggregate query-relevant information, performing retrieval at the final layer. In Stage 2, only the retrieved visual tokens Retrieved Vision Tokens are supplied to the model to generate the final answer.

Appendix B Additional Details

B.1 Training Data Filtering

The original MIRAGE fine-tuning set aggregates multiple open-source VQA datasets, many of which were already seen during Qwen3VL pretraining. We observe that on a substantial fraction of these examples, Qwen3VL achieves lower generation loss when conditioned on the full set of distractor images than on the target ground-truth image alone, suggesting that the model has memorized the underlying QA pairs and bypasses the intended retrieval task. To address this, we apply three filters to the MIRAGE-augmented dataset (in which images from different QAs are combined into a single multi-image input):

•

Memorization filter. We discard examples where the generation loss conditioned on the target image alone is higher than that conditioned on the full image set, as such examples no longer require retrieval to be answered.

•

Difficulty filter. We rank remaining examples by the average attention score between the question query tokens and the target image key tokens, and retain those with higher scores. In a controlled experiment, we find that easier retrieval examples (higher attention scores on the target image) lead to a stronger learned retriever than harder examples.

•

Multi-image filter. We remove single-image QA examples (which lack distractor images for retrieval) and OCR-style queries (which test text recognition rather than visual recognition).

媒体内容 · 前往原文查看

Table 12: The influence of training dataset. “w/o filtered” means directly sampling 70,686 examples from the original MIRAGE fine-tuning set without filtering, and using them to train ReToken. “w/ filtered” is our default setting.

Method

Qwen3VL-8B Bai et al. (2025) 87.8 82.0 77.9 74.7 69.2 64.0 58.6 55.7

w/o filtered 87.8 85.2 83.1 80.8 78.1 75.5 69.3 62.7

w/ filtered 87.8 85.9 84.4 82.5 80.7 76.8 72.0 67.7

After filtering, there are 70,686 examples in our final training dataset. We note that our filters are applied only to the MIRAGE training split; the Visual Haystacks evaluation split and video datasets are left untouched. The filtering signals (generation loss and attention scores) are computed from the frozen Qwen3VL-8B on training examples only, with no access to evaluation labels or examples. Tab. 12 shows the influence of filtering.

B.2 Chain-of-Thought (CoT) Retrieval Baseline

B.3 Key or Value? Training Comparison

Figure 6: Training Comparison. The ReToken Value variant (a) converges much faster in retrieval loss, and (b) learns a substantially larger retrieval score gap between relevant and irrelevant images.

Fig. 6 shows the retrieval loss and retrieval score gap across training steps with the VLM frozen and only ReToken trained: Value yields a clear advantage.

Appendix C Design Choice

C.1 Multiple Tokens

Tab. 13 reports results for training 3 ReTokens on the retrieval task. We append the 3 ReTokens sequentially, and compute the retrieval score by computing the cosine similarity between each projected ReToken and the averaged image value feature. We then average the resulting logits to obtain the final score.

媒体内容 · 前往原文查看

Table 13: Multiple tokens perform on par with a single token. Qwen3VL-8B.

Method

3 tokens 87.8 85.5 84.3 82.5 82.0 77.6 72.1 67.9

1 token 87.8 85.9 84.4 82.5 80.7 76.8 72.0 67.7

Performance is on par overall, with small fluctuations in both directions. One reason may lie in our training recipe: the multi-token design treats the embeddings equally, since all three are supervised through one averaged score, nothing encourages them to specialize, so the tokens likely converge on a similar solution. A promising direction could be to empower specialization of different tokens so that a token is more pronounced for its specialized domain. But this is beyond the scope of this work and we leave it for future exploration.

C.2 Query Composition or visual Summarization

Since we append the ReToken to the input, it can attend to both the images and the query, allowing it to serve as both a visual summarization token and a query summarization token. To better understand the role of the ReToken, we design an ablation in which the ReToken is only allowed to attend to the query tokens. In this setting, we perform two forward streams. In stream A, we encode the vision tokens and cache their KV. In stream B, we process only the query and the ReToken, so that neither the query token nor the ReToken can attend to the visual part. This way, the ReToken sees only the query tokens and learns to summarize what we want to retrieve based on the input query alone. We then compute the retrieval score by computing the cosine similarity between the projected ReToken from the last layer of stream B and the averaged visual value feature from the last layer of stream A.

媒体内容 · 前往原文查看

Table 14: Allowing the ReToken to attend to images yields better results. Question tokens also skip the visual tokens in the “skip images” setting. The VLM is frozen.

Method

Qwen3VL-8B Bai et al. (2025) 87.8 82.0 77.9 74.7 69.2 64.0 58.6 55.7

skip images 87.8 83.4 78.9 76.2 73.7 68.5 63.1 62.0

attend images 87.8 85.9 84.4 82.5 80.7 76.8 72.0 67.7

Tab. 14 reports the results with and without attention to the images. The results show that skipping the visual tokens entirely still yields decent performance, but underperforms the variant that attends to the images. This suggests that when the ReToken can understand what happens in the input video/images, it achieves better retrieval results.