# OmniPack：面向高效全模态大语言模型的统一 token 压缩框架 - 来源：HuggingFace Daily Papers（社区热门论文） - 发布时间：2026-08-04 08:00 - AIHOT 分数：48 - AIHOT 链接：https://aihot.virxact.com/items/cmsfhv0y71tjkro2e7wj14tq9 - 原文链接：https://arxiv.org/abs/2608.03812 ## AI 摘要 OmniPack 提出一种免训练的两阶段 token 压缩框架，在 LLM 前进行结构化压缩、在 LLM 内进行语义精炼，以解决全模态大语言模型处理长序列时的高计算开销问题。 ## 正文 Wanshun Su Feihu Liu Ziwen Yu Yan Min Zhuoran Zhang Qixun Wang Haotian Wang Shixuan Liu Peng Wu Chengfu Huo Liang Ding Northwestern Polytechnical University Abstract Omni-modal large language models (Omni-LLMs) have achieved remarkable performance on audio-visual understanding tasks, but processing long and highly redundant visual and audio token sequences incurs substantial computational overhead, demanding aggressive token compression for efficient deployment. Existing methods often degrade at low token budgets: pre-LLM compression may discard structurally important and globally distributed evidence, whereas inner-LLM compression often underexploits query-conditioned audio-visual collaboration. To address these limitations, we propose OmniPack, a training-free framework that coordinates structural compression before the LLM with task-relevant semantic refinement within the LLM. Before the LLM, OmniPack removes structural redundancy through modality-specific importance, global coverage, and similarity-aware merging. After sufficient multimodal interaction, it further consolidates diverse, task-relevant representations through textual guidance and audio-visual collaboration. Extensive experiments on five benchmarks with three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance–efficiency trade-off across diverse retention ratios, outperforming all existing methods. Notably, on Qwen2.5-Omni-7B, OmniPack preserves 98.0% of the original performance while reducing FLOPs to 16.7%, and still retains 92.9% of the original performance with only 6.8% of the original FLOPs. 1 Introduction The rapid development of omni-modal large language models (Omni-LLMs) [46, 49, 45], which jointly model visual, audio, and textual inputs, has significantly advanced unified audio-visual understanding [11, 13, 21, 30, 32, 40, 41]. However, densely tokenized visual and audio inputs produce substantially longer sequences than text, incurring considerable computation and memory overhead during inference. Efficient deployment therefore requires aggressive token compression that removes redundancy without losing task-relevant multimodal evidence. Figure 1: Performance comparison on Qwen2.5-Omni-7B across five benchmarks. SEATS variants and OmniPack further reduce tokens from 15% to 7.5%, with OmniPack achieving the best overall performance. Audio-visual token compression [34, 7] has recently emerged as a practical solution for reducing the prohibitive computational cost of long multimodal sequences. Existing Omni-LLM compression methods adopt different compression schedules. Some reduce audio-visual tokens only before the LLM [34, 14, 7, 6], whereas others progressively compress tokens both before and within the LLM [26, 42]. Despite their effectiveness, existing approaches still face two limitations under aggressive token budgets. First, existing methods [34, 7] do not sufficiently model the long-range spatiotemporal structure of audio-visual inputs. Task-relevant evidence may be sparsely distributed across distant temporal segments, abrupt visual changes, and transient acoustic events. Compression based primarily on local importance or token similarity may therefore overlook globally distributed evidence, particularly in long videos. Moreover, directly discarding low-scoring tokens can cause irreversible information loss, as individually less salient tokens may still provide complementary context. Second, existing methods do not sufficiently align audio-visual collaboration with the semantic maturity of multimodal representations. Some methods use one modality to guide the compression of another before the LLM [34, 7]. At this stage, independently encoded audio and visual representations have undergone limited semantic interaction, such that cross-modal guidance may rely on insufficiently aligned features. Although progressive methods further compress tokens within the LLM [26, 42], their inner-LLM stages primarily rely on textual guidance without explicitly modeling the collaborative interactions between audio and visual tokens. These observations motivate a stage-specialized progressive strategy: pre-LLM compression should exploit modality-specific spatiotemporal structure, whereas inner-LLM compression should leverage query-conditioned audio-visual interactions established within the LLM. Based on this insight, we propose OmniPack, a training-free two-stage framework that progressively compresses tokens before and within the LLM. Pre-LLM compression exploits modality-specific spatiotemporal structure to remove large-scale redundancy while preserving important evidence and global coverage, providing early computational savings. After sufficient multimodal interaction, inner-LLM compression further refines the retained representations using textual guidance and audio-visual collaboration. Together, the two stages coordinate structural compression with semantic refinement, enabling aggressive token compression while maintaining multimodal understanding performance. Extensive experiments demonstrate that OmniPack generalizes effectively across three Omni-LLM backbones and five benchmarks. At a final token retention ratio of 7.5%, OmniPack reduces FLOPs by 10.0 and achieves a 4.5 prefill speedup while preserving 95.6% of the original performance. Figure 1 further shows that OmniPack consistently offers the best performance–efficiency trade-off among existing token compression methods. • We identify limitations of existing Omni-LLM compression under aggressive token budgets: inadequate long-range audio-visual structure modeling and insufficient exploitation of query-conditioned cross-modal evidence. • We propose OmniPack, a training-free progressive framework that combines modality-specific pre-LLM compression with query-conditioned inner-LLM compression, while merging removed information into retained representatives. • Experiments across multiple Omni-LLMs and benchmarks show that OmniPack preserves strong multimodal understanding while substantially reducing computation and accelerating inference. 2 Related Work 2.1 Omni-modal Large Language Models Multimodal large language models (MLLMs) [16, 25, 29] are rapidly evolving from VLMs [1, 39, 37, 38, 51] toward Omni-LLMs [36, 5, 18, 4]. Compared with earlier visual-centric models, Omni-LLMs can seamlessly process and understand text, images, videos, and audio within a unified framework, enabling richer multimodal interactions and broader applications, including audio-visual understanding [36, 4], real-time conversational assistants [5], and multimodal agents [22]. Representative models, such as Gemini 3.1 Pro [16], GPT-4o [18], Qwen2.5-Omni [44], and MiniCPM-o [5], have demonstrated strong capabilities in unified multimodal understanding and generation. Their ability to jointly model audio and visual cues is particularly valuable for understanding complex, dynamic, and temporally evolving real-world scenarios that require cross-modal reasoning. However, dense audio-visual inputs introduce substantial token and computational overhead. Therefore, efficient inference is essential for the practical deployment of Omni-LLMs. Figure 2: Overview of the OmniPack framework. Before the LLM, modality-specific compression performs importance selection, coverage selection, and similarity-aware token merging. Within the LLM, query-conditioned compression is applied after the -th Transformer block. The recycle pool and retention pool denote unselected and selected tokens. 2.2 Token Compression in Omni-LLMs Token compression has been extensively studied for image [2, 43, 31, 47], video [33, 28, 10, 8], and audio [20, 23] inputs to reduce multimodal redundancy and improve inference efficiency. With the rapid development of Omni-LLMs, recent studies have extended token compression to joint audio-visual inputs [34, 7, 15]. OmniZip [34] introduces audio-guided video token compression, whereas OmniSIFT [7] combines spatio-temporal video compression with vision-guided audio selection. EchoingPixels [15], ContextGuard [19], and OmniRefine [6] investigate joint audio-visual contextualization, cross-modal recoverability, and aligned compression units, respectively. OmniDrop [26] and SEATS [42] explore progressive, query-guided token pruning within the LLM. Despite these advances, existing methods either introduce cross-modal guidance before sufficient audio–visual interaction has been established or perform inner-LLM pruning without explicit audio-visual collaboration. As a result, they struggle to jointly preserve modality-specific structure and exploit task-relevant semantics, particularly under aggressive compression. This gap motivates a unified framework that coordinates structure-aware pre-LLM compression with query-conditioned audio–visual compression inside the LLM. 3 Method 3.1 Preliminary Given a video and its aligned audio , the modality-specific encoder–projector pipelines map them into the LLM embedding space, producing visual tokens and audio tokens . Together with the text tokens, they form the multimodal input sequence to the shared LLM backbone. For the encoder tokens of modality , we retain tokens before the LLM: (1) where is the number of input tokens, is the pre-LLM retention ratio, and is the corresponding token budget for modality . Let denote the number of modality tokens passed to the LLM. Given the inner-LLM retention ratio , we further retain the following number of tokens for modality : (2) All text tokens remain unchanged throughout compression. 3.2 Overview of OmniPack As shown in Figure 2, OmniPack combines modality-specific pre-LLM compression with query-conditioned inner-LLM compression. Pre-LLM compression removes structural redundancy while preserving salient and representative information, whereas inner-LLM compression refines retained tokens using textual relevance, audio-visual collaboration, and within-modality representativeness. Together, they shift compression from structural cues to task-conditioned semantics with minimal information loss. 3.3 Modality-Specific Pre-LLM Compression Pre-LLM compression operates independently on each modality, since cross-modal semantics have not yet sufficiently emerged, as illustrated in Figure 2. Importance Selection. We first identify dominant tokens using encoder attention and modality-specific structural variation. Attention statistics from the final modality encoder provide a base estimate of token centrality: (3) Here, and are the query and key matrices for modality , is the hidden dimension, and is the resulting attention matrix. We measure the mean attention received by each token from all other modality tokens, yielding the attention-centrality vector . Attention alone may overlook transient structural changes, so we refine it using modality-specific variation cues. For video, let denote the token at frame and spatial position . We define adjacent-frame variation and spatial distinctiveness as: (4) Here, is the number of spatial tokens per frame. The two cues capture inter-frame changes and spatial distinctiveness, respectively. For audio, let denote the audio token at temporal index . Adjacent-token variation is defined as: (5) where a larger indicates a stronger acoustic boundary or transient event. After normalization, the modality-specific structural cues are added to the attention-centrality vector to obtain the final modality-specific importance vector , where denotes the final importance score of token . We further denote its normalized score as . Since importance-only selection may overconcentrate tokens around a few salient regions, we allocate only part of the pre-LLM token budget to dominant tokens: (6) where denotes the fraction of the token budget allocated to importance selection. The top- tokens ranked by form the dominant set , while the remaining tokens are determined by coverage selection. Coverage Selection. Importance selection may concentrate tokens around a few salient events and leave distant content underrepresented. We therefore select additional representatives using both feature and positional distances. For modality , we define a joint distance between tokens and as: (7) where denotes the normalized temporal distance for audio or spatiotemporal distance for video. The joint distance captures both representation similarity and structural proximity, preventing distant tokens with similar features from being considered equivalent. The parameter controls the contribution of positional distance. Based on this joint distance, we apply DPC-KNN [9, 27] to identify tokens that are both representative of local neighborhoods and separated from other representative regions. The detailed DPC-KNN formulation is provided in the Appendix. The top-ranked candidates outside fill the remaining token budget to form . The final pre-LLM retained set is: (8) Thus, importance selection preserves salient evidence, while coverage selection maintains broad spatiotemporal coverage. Similarity-Aware Token Merging. Instead of directly discarding unselected tokens, we transfer their information to retained representatives. Let denote the set of unselected token indices. For each unselected token and retained token , we compute their affinity and determine the corresponding merge target: (9) Here, jointly considers feature similarity, positional proximity, and the normalized importance of retained token . The parameter controls the modality-specific positional constraint, while controls the contribution of representative importance. The assignment identifies the retained representative that receives unselected token . Each retained token is then updated by aggregating the unselected tokens assigned to it: (10) Here, assigns larger weights to more important tokens, enabling each representative to aggregate nearby redundant information while mitigating information loss. The updated representations form , which is fed into the LLM with the text tokens. 3.4 Query-Conditioned Inner-LLM Compression After the first Transformer blocks, the pre-LLM compressed sequence is transformed into hidden states , where and . We then perform query-conditioned audio-visual collaborative compression on these representations. Let denote the text hidden states. We mean-pool into a global query representation and the other-modality states into a prototype . The relevance score of token is: (11) Here, denotes modality-wise min–max normalization, and the three terms measure textual relevance, audio-visual collaboration, and within-modality representativeness, respectively. We initialize the retained set with the highest-scoring token as: . For each unselected token, we measure its semantic diversity from the retained set by . We then select and add the token that best balances relevance and diversity: (12) This process continues until . Each unselected hidden state is assigned to its most cosine-similar retained representative and incorporated through relevance-modulated similarity-weighted averaging. The resulting retained states form with tokens. The compressed audio-visual tokens and all uncompressed text tokens are then passed through the remaining Transformer blocks. 4 Experiment 4.1 Experimental Settings Benchmarks. We evaluate OmniPack on five widely used audio-visual understanding benchmarks: AVUT [48], WorldSense [17], DailyOmni [52], VideoMME [12], and LVOmniBench [35]. Together, they cover audio-centric and video-centric understanding, short- and long-form videos, and tasks ranging from perception to reasoning. Comparison Methods. We compare OmniPack with representative training-free baselines, including image-based FastV [3] and VisionZip [47], video-based FastVID [28] and VidCom2 [24], omni-modal OmniZip [34], OmniSIFT [7], SEATS [42], and Random. Following prior work [42], FastV-om and VisionZip-om extend FastV and VisionZip to both video and audio tokens. OmniSIFT retains only its inference-time compression strategy without alignment training. SEATS uses the original configuration, while SEATS disables late-layer token removal; both match the final modality-specific retention ratios and are evaluated only on the 28-layer Qwen2.5-Omni-7B and MiniCPM-o-2.6 backbones. Implementation Details. We implement OmniPack on Qwen2.5-Omni-3B/7B [44] and MiniCPM-o-2.6 [49] using NVIDIA H20 GPUs. The maximum number of frames is set to 128 for AVUT, WorldSense, and DailyOmni, and 768 for VideoMME and LVOmniBench. Following prior work [34, 42], we evaluate four pre-LLM retention ratios—25%, 20%, 15%, and 10%—using model-specific visual and audio allocations. For inner-LLM compression, we retain 50% of the pre-LLM tokens at layer 18 for Qwen2.5-Omni-7B and MiniCPM-o-2.6, and layer 26 for Qwen2.5-Omni-3B. We set =(0.25,0.35) and =0.20 for coverage selection. For token merging, we use =(0.10,0.05) and =0.10. All evaluations are conducted with LMMs-Eval [50]. Additional details are provided in the Appendix. 媒体内容 · 前往原文查看 Method Efficiency Benchmark Performance Average FLOPs (T) Relative FLOPs Retention Ratio AVUT World Sense Daily Omni MME LVOmni Avg. Score Relative Perf. (%) \rowcolorgray!18 Qwen2.5-Omni-7B 73.2 100.0% 100% / 100% 64.0 46.6 63.9 64.4 34.2 54.6 100.0 Random 14.9 20.4% 25% / – 58.7 43.1 57.1 65.1 32.1 51.2 93.8 FastV 19.1 26.1% 100% / 25% 54.9 42.0 56.6 60.7 33.1 49.5 90.7 FastV-om 19.1 26.1% 100% / 25% 59.3 43.3 58.6 64.0 32.6 51.6 94.5 VisionZip 14.9 20.4% 25% / – 56.8 44.0 59.3 63.4 31.7 51.0 93.4 VisionZip-om 14.9 20.4% 25% / – 58.7 44.7 58.2 65.1 32.9 51.9 95.1 FastVID 13.6 18.5% 25% / – 58.0 42.8 58.2 60.1 33.2 50.5 92.5 VidCom2 14.9 20.4% 25% / – 58.8 43.3 57.7 63.7 33.1 51.3 94.0 OmniZip 14.9 20.4% 25% / – 56.9 42.6 55.9 65.9 35.0 51.3 93.8 OmniSIFT 14.9 20.4% 25% / – 58.4 42.1 58.0 65.9 32.5 51.4 94.1 SEATS 11.6 15.8% 25% / 12.5% 60.0 45.0 59.7 66.3 33.5 52.9 96.7 SEATS 12.5 17.1% 25% / 12.5% 60.1 45.0 59.8 66.4 33.9 53.0 97.1 OmniPack (w/o M) 14.9 20.4% 25% / – 60.7 45.3 60.3 66.0 35.6 53.6 98.2 \rowcolorblue!8 OmniPack 12.2 16.7% 25% / 12.5% 61.0 45.3 60.6 65.7 35.0 53.5 98.0 Random 11.8 16.2% 20% / – 58.6 42.7 56.7 64.0 34.3 51.3 94.0 FastV-om 16.2 22.2% 100% / 20% 56.2 41.6 56.1 62.3 31.9 49.6 90.8 VisionZip 11.8 16.2% 20% / – 56.8 43.4 57.7 61.6 31.7 50.2 91.9 VisionZip-om 11.8 16.2% 20% / – 57.1 44.0 58.0 64.3 32.6 51.2 93.8 FastVID 11.2 15.3% 20% / – 55.9 42.2 56.4 59.1 32.5 49.2 90.1 VidCom2 11.8 16.2% 20% / – 55.6 41.7 55.1 59.9 32.4 48.9 89.6 OmniSIFT 11.8 16.2% 20% / – 56.5 42.2 54.3 64.7 33.3 50.2 91.9 SEATS 9.2 12.6% 20% / 10% 59.1 44.2 58.6 65.3 33.9 52.2 95.6 SEATS 10.0 13.6% 20% / 10% 59.1 44.5 58.9 65.4 33.8 52.3 95.7 OmniPack (w/o M) 11.8 16.2% 20% / – 60.0 45.0 58.7 65.0 35.7 52.9 96.9 \rowcolorblue!8 OmniPack 9.8 13.3% 20% / 10% 60.0 44.8 58.6 65.3 35.7 52.9 96.9 Random 8.9 12.1% 15% / – 56.4 41.1 54.8 63.0 33.3 49.7 91.0 FastV-om 13.5 18.4% 100% / 15% 51.2 38.5 54.7 57.5 32.6 46.9 85.9 VisionZip-om 8.9 12.1% 15% / – 56.1 43.0 56.6 61.6 32.1 49.9 91.4 OmniSIFT 8.9 12.1% 15% / – 53.9 41.1 54.8 62.9 32.3 49.0 89.7 SEATS 6.9 9.5% 15% / 7.5% 57.7 43.3 57.7 63.6 32.8 51.0 93.4 SEATS 7.5 10.3% 15% / 7.5% 57.7 43.4 57.6 63.8 32.5 51.0 93.4 OmniPack (w/o M) 8.9 12.1% 15% / – 57.8 44.6 58.2 64.7 34.9 52.0 95.2 \rowcolorblue!8 OmniPack 7.3 10.0% 15% / 7.5% 58.1 44.6 58.2 64.7 35.3 52.2 95.6 Random 6.0 8.2% 10% / – 53.4 39.3 52.0 61.1 33.7 47.9 87.7 VisionZip-om 6.0 8.2% 10% / – 53.6 41.4 55.0 60.0 33.4 48.7 89.2 OmniSIFT 6.0 8.2% 10% / – 51.8 38.8 51.4 61.3 32.8 47.2 86.4 SEATS 4.7 6.5% 10% / 5% 56.7 42.0 55.3 62.0 32.9 49.8 91.2 SEATS 5.1 7.0% 10% / 5% 56.7 42.3 55.5 61.7 32.8 49.8 91.2 OmniPack (w/o M) 6.0 8.2% 10% / – 56.8 42.9 56.1 63.5 34.8 50.8 93.0 \rowcolorblue!8 OmniPack 5.0 6.8% 10% / 5% 56.4 42.7 56.1 63.6 34.8 50.7 92.9 Table 1: Comparison with state-of-the-art methods on Qwen2.5-Omni-7B. The A%/B% denotes the retention ratios before and after inner-LLM compression, respectively. “–” indicates that inner-LLM compression is not applied. Best results are in bold, and second-best results are underlined. “OmniPack (w/o M)” denotes our method without inner-LLM compression. 4.2 Main Results State-of-the-Art Performance. As shown in Table 1, our approach achieves state-of-the-art performance across all retention settings. At a 25% retention ratio, OmniPack (w/o M) preserves 98.2% of the original performance. When the retention ratio is reduced to 15%, VisionZip-om and OmniSIFT preserve 91.4% and 89.7% of the original performance, respectively, whereas OmniPack (w/o M) retains 95.2%. Under extreme compression at a 10% retention ratio, OmniPack (w/o M) still preserves 93.0% of the original performance using only 8.2% of the original FLOPs, outperforming both SEATS variants and VisionZip-om. The query-conditioned inner-LLM compression further reduces computation while maintaining comparable performance. At the 15%/7.5% and 10%/5% retention settings, OmniPack preserves 95.6% and 92.9% of the original performance while using only 10.0% and 6.8% of the original FLOPs, respectively. These results demonstrate a strong performance–efficiency trade-off under extremely low token budgets. Different Backbones. 媒体内容 · 前往原文查看 Method Efficiency Benchmark Performance Average FLOPs (T) Relative FLOPs Retention Ratio AVUT World Sense Daily Omni MME LVOmni Bench Avg. Score Relative Perf. (%) \rowcolorgray!18 Qwen2.5-Omni-3B 37.4 100.0% 100% / 100% 62.5 46.5 61.0 62.5 35.2 53.5 100.0 Random 3.9 10.5% 15% / – 54.4 40.4 50.5 59.4 32.6 47.5 88.8 FastV-om 5.8 15.5% 100% / 15% 51.0 41.1 50.5 53.4 32.6 45.7 85.4 VisionZip-om 3.9 10.5% 15% / – 54.2 42.9 54.1 59.6 32.3 48.6 90.8 OmniSIFT 3.9 10.5% 15% / – 52.9 39.4 49.4 59.5 33.9 47.0 87.9 OmniPack (w/o M) 3.9 10.5% 15% / – 57.1 43.8 53.2 59.8 34.0 49.6 92.7 \rowcolorblue!8 OmniPack 3.4 9.0% 15% / 7.5% 57.4 43.9 52.7 60.2 33.8 49.6 92.7 \rowcolorgray!18 MiniCPM-o-2.6 73.2 100.0% 100% / 100% 53.2 40.8 52.2 62.0 29.5 47.5 100.0 Random 8.9 12.1% 15% / – 47.7 38.5 51.0 61.4 36.0 46.9 98.7 FastV-om 13.5 18.4% 100% / 15% 43.5 36.1 43.8 51.6 32.7 41.5 87.4 VisionZip-om 8.9 12.1% 15% / – 49.4 38.7 52.0 60.2 35.5 47.2 99.4 OmniSIFT 8.9 12.1% 15% / – 48.4 37.9 50.6 59.6 34.5 46.2 97.3 SEATS 6.9 9.5% 15% / 7.5% 49.8 38.8 50.5 60.0 34.4 46.7 98.3 SEATS 7.5 10.3% 15% / 7.5% 49.8 39.1 49.1 60.4 34.6 46.6 98.1 OmniPack (w/o M) 8.9 12.1% 15% / – 51.3 40.0 52.0 61.1 34.3 47.7 100.4 \rowcolorblue!8 OmniPack 7.3 10.0% 15% / 7.5% 51.2 39.8 52.0 61.7 34.6 47.9 100.8 Table 2: Cross-backbone comparison on Qwen2.5-Omni-3B and MiniCPM-o-2.6. Table 2 further evaluates our method on Qwen2.5-Omni-3B and MiniCPM-o-2.6, covering different model scales and architectures. On Qwen2.5-Omni-3B, OmniPack preserves 92.7% of the original performance at a 15%/7.5% retention ratio while using only 9.0% of the original FLOPs. On MiniCPM-o-2.6, OmniPack achieves 100.8% of the original performance under the same retention setting, outperforming existing compression methods while using only 10.0% of the original FLOPs. 4.3 Efficiency Analysis 媒体内容 · 前往原文查看 Method Token Retention Pre-LLM / Final FLOPs (T) Prefill (sec.) Avg. Score \rowcolorgray!18 Vanilla 100% 73.2 1.512 54.6 FastV-om 100%/25% 19.1 (3.8) 0.415 (3.6) 51.6 OmniZip 25%/– 14.9 (4.9) 0.360 (4.2) 51.3 VisionZip-om 25%/– 14.9 (4.9) 0.382 (4.0) 51.9 OmniSIFT 25%/– 14.9 (4.9) 0.434 (3.5) 51.4 SEATS 25%/12.5% 11.6 (6.3) 0.348 (4.3) 52.9 SEATS 25%/12.5% 12.5 (5.9) 0.375 (4.0) 53.0 OmniPack (w/o M) 25%/– 14.9 (4.9) 0.396 (3.8) 53.6 OmniPack 25%/12.5% 12.2 (6.0) 0.504 (3.0) 53.5 OmniPack 20%/10% 9.8 (7.5) 0.424 (3.6) 52.9 \rowcolorblue!8 OmniPack 15%/7.5% 7.3 (10.0) 0.339 (4.5) 52.2 Table 3: Efficiency analysis on Qwen2.5-Omni-7B using an NVIDIA H20 GPU. As shown in Table 3, at the 25% retention ratio, OmniPack (w/o M) preserves 98.2% of the original performance while reducing FLOPs by 4.9. Crucially, at the 15%/7.5% retention ratio, OmniPack preserves 95.6% of the original performance, outperforming all pre-LLM compression methods at 25% retention ratio despite retaining substantially fewer tokens. This setting reduces FLOPs by 10.0 and achieves a 4.5 prefill speedup, demonstrating a favorable performance–efficiency trade-off. 4.4 Ablation Studies We conduct ablation studies on our pre-LLM compression strategy, our inner-LLM compression strategy, and their compatibility. All ablation experiments are conducted on Qwen2.5-Omni-7B. Pre-LLM compression strategy ablations are evaluated at a 15% retention ratio without inner-LLM compression. Unless otherwise specified, the remaining ablations use a 15%/7.5% retention setting. Pre-LLM Compression Strategy. 媒体内容 · 前往原文查看 Importance Coverage Merge World Sense MME LVOmni Bench ✓ 41.7 62.0 32.7 ✓ 43.1 63.3 34.7 ✓ 41.7 63.9 33.7 ✓ ✓ 44.4 64.5 33.3 ✓ ✓ 41.9 62.4 32.1 ✓ ✓ 44.2 62.7 34.1 ✓ ✓ ✓ 44.6 64.7 34.9 Table 4: Ablation of the pre-LLM compression components. 媒体内容 · 前往原文查看 Method AVUT World Sense Daily Omni w/o A-V Collaboration 57.8 44.5 57.8 w/ A-V Collaboration 58.1 44.6 58.2 Table 5: Ablation of audio-visual collaboration. 媒体内容 · 前往原文查看 Guidance Strategy AVUT World Sense Daily Omni General Query 57.7 44.6 58.1 Last-Token Attention 57.9 44.5 58.2 OmniPack (Text-Aware Guidance) 58.1 44.6 58.2 Table 6: Ablation of different text-guidance mechanisms. 媒体内容 · 前往原文查看 ID Pre-LLM Inner-LLM AVUT World Sense Daily Omni MME LVOmni Bench Avg. Score Vanilla – – 64.0 46.6 63.9 64.4 34.2 54.6 1 VisionZip-om Ours 56.2 42.9 56.6 61.7 31.1 49.7 2 OmniSIFT Ours 53.5 41.0 53.7 64.0 32.3 48.9 3 SEATS Ours 57.4 43.1 57.4 63.7 33.1 50.9 4 Ours SEATS 58.0 44.7 57.8 64.7 33.9 51.8 OmniPack Ours Ours 58.1 44.6 58.2 64.7 35.3 52.2 Table 7: Compatibility of pre-LLM and inner-LLM compression strategies. As shown in Table 4, coverage selection performs best among the individual components on WorldSense and LVOmniBench. Relative to importance selection, it improves the two benchmarks by approximately 3.4% and 6.1%, respectively. Relative to token merging, the gains are 3.4% and 3.0%. Combining importance and coverage further improves WorldSense and VideoMME over coverage alone by approximately 3.0% and 1.9%. Incorporating all three components achieves the best performance across the three benchmarks, including a further 4.8% gain on LVOmniBench over the importance-and-coverage configuration. These results demonstrate the complementary roles of the three components in pre-LLM compression. Inner-LLM Compression Strategy. Figure 3: Effect of the inner-LLM compression layer. Figure 3 and Tables 5 and 6 analyze the key mechanisms of our inner-LLM compression strategy. Applying the strategy too early limits task-conditioned multimodal interaction, whereas applying it too late reduces the computational benefit. Layer 18 achieves the best overall performance and is therefore adopted by default. Audio-visual collaboration consistently outperforms independent compression by better preserving complementary cross-modal information. Moreover, compared with question-agnostic general query and last-token guidance, the proposed text-aware guidance aggregates richer semantic cues from the full textual query and achieves the best overall performance. Compression Strategies Compatibility. Table 7 evaluates the compatibility of different pre-LLM and inner-LLM compression strategies. Pairing our inner-LLM compression with VisionZip-om, OmniSIFT, or SEATS consistently underperforms the full OmniPack, while replacing our inner-LLM compression with SEATS also lowers the average score and degrades three of the five benchmarks. These results demonstrate that the two stage-specific strategies are complementary and work effectively together. 5 Conclusion We present OmniPack, a training-free token compression framework for efficient Omni-LLMs. Rather than restricting compression to a single location, OmniPack progressively exploits modality-specific structural information before the LLM and task-conditioned semantic information after sufficient multimodal interaction within the LLM. This progressive design enables substantially more aggressive compression while effectively preserving downstream performance. Extensive experiments on five audio-visual understanding benchmarks and three Omni-LLM backbones demonstrate that OmniPack consistently achieves the best performance–efficiency trade-off among existing methods across diverse token budgets. References Bai et al. [2025] Shuai Bai, Yuxuan Cai, Ruizhe Chen, Keqin Chen, Xionghui Chen, Zesen Cheng, Lianghao Deng, Wei Ding, Chang Gao, Chunjiang Ge, et al. Qwen3-vl technical report. arXiv preprint arXiv:2511.21631, 2025. Bolya et al. [2022] Daniel Bolya, Cheng-Yang Fu, Xiaoliang Dai, Peizhao Zhang, Christoph Feichtenhofer, and Judy Hoffman. Token merging: Your vit but faster. arXiv preprint arXiv:2210.09461, 2022. Chen et al. [2024] Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In European Conference on Computer Vision, 2024. Chen et al. [2025] Xinlong Chen, Yue Ding, Weihong Lin, Jingyun Hua, Linli Yao, Yang Shi, Bozhou Li, Yuanxing Zhang, Qiang Liu, Pengfei Wan, et al. Avocado: An audiovisual video captioner driven by temporal orchestration. arXiv preprint arXiv:2510.10395, 2025. Cui et al. [2026] Junbo Cui, Bokai Xu, Chongyi Wang, Tianyu Yu, Weiyue Sun, Yingjing Xu, Tianran Wang, Zhihui He, Wenshuo Ma, Tianchi Cai, et al. Minicpm-o 4.5: Towards real-time full-duplex omni-modal interaction. arXiv preprint arXiv:2604.27393, 2026. Deng et al. [2026] Yuchen Deng, Zidang Cai, Hai-Tao Zheng, Jie Wang, Feidiao Yang, and Yuxing Han. Omnirefine: Alignment-aware cooperative compression for efficient omnimodal large language models. arXiv preprint arXiv:2605.12056, 2026. Ding et al. [2026] Yue Ding, Yiyan Ji, Jungang Li, Xuyang Liu, Xinlong Chen, Junfei Wu, Bozhou Li, Bohan Zeng, Yang Shi, Yushuo Guan, et al. Omnisift: Modality-asymmetric token compression for efficient omni-modal large language models. arXiv preprint arXiv:2602.04804, 2026. Du et al. [2026] Junhao Du, Jialong Xue, Anqi Li, Jincheng Dai, and Guo Lu. Unified spatiotemporal token compression for video-llms at ultra-low retention. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026. Du et al. [2016] Mingjing Du, Shifei Ding, and Hongjie Jia. Study on density peaks clustering based on k-nearest neighbors and principal component analysis. Knowledge-Based Systems, 2016. Fan et al. [2026] Ziyang Fan, Keyu Chen, Ruilong Xing, Yulin Li, Li Jiang, and Zhuotao Tian. Flashvid: Efficient video large language models via training-free tree-based spatiotemporal token merging. arXiv preprint arXiv:2602.08024, 2026. Fu et al. [2024] Chaoyou Fu, Haojia Lin, Zuwei Long, Yunhang Shen, Yuhang Dai, Meng Zhao, Yi-Fan Zhang, Shaoqi Dong, Yangze Li, Xiong Wang, et al. Vita: Towards open-source interactive omni multimodal llm. arXiv preprint arXiv:2408.05211, 2024. Fu et al. [2025] Chaoyou Fu, Yuhan Dai, Yongdong Luo, Lei Li, Shuhuai Ren, Renrui Zhang, Zihan Wang, Chenyu Zhou, Yunhang Shen, Mengdan Zhang, et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 24108–24118, 2025. Ge et al. [2025] Yuying Ge, Yixiao Ge, Chen Li, Teng Wang, Junfu Pu, Yizhuo Li, Lu Qiu, Jin Ma, Lisheng Duan, Xinyu Zuo, et al. Arc-hunyuan-video-7b: Structured video comprehension of real-world shorts. arXiv preprint arXiv:2507.20939, 2025. Gong et al. [2025] Chao Gong, Depeng Wang, Zhipeng Wei, Ya Guo, Huijia Zhu, and Jingjing Chen. Echoingpixels: Cross-modal adaptive token reduction for efficient audio-visual llms. arXiv preprint arXiv:2512.10324, 2025. Gong et al. [2026] Chao Gong, Depeng Wang, Zhipeng Wei, Ya Guo, Huijia Zhu, and Jingjing Chen. Echoingpixels: Aliasing-resistant joint token reduction for audio-visual llms. In Forty-third International Conference on Machine Learning, 2026. Google DeepMind [2026] Google DeepMind. Gemini 3.1 pro model card. https://deepmind.google/models/model-cards/gemini-3-1-pro/, 2026. Accessed: 2026-06. Hong et al. [2025] Jack Hong, Shilin Yan, Jiayin Cai, Xiaolong Jiang, Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal llms. arXiv preprint arXiv:2502.04326, 2025. Hurst et al. [2024] Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024. Jin et al. [2026] Hongbo Jin, Chi Wang, Haoran Tang, Zhongjing Du, Xu Jiang, Jingqi Tian, Qiaoman Zhang, and Jiayu Ding. Contextguard: Structured self-auditing for context learning in language models. arXiv preprint arXiv:2605.26827, 2026. Lee and Lee [2025] Taehan Lee and Hyukjun Lee. Token pruning in audio transformers: Optimizing performance and decoding patch importance. arXiv preprint arXiv:2504.01690, 2025. Li et al. [2025] Caorui Li, Yu Chen, Yiyan Ji, Jin Xu, Zhenyu Cui, Shihao Li, Yuanxing Zhang, Wentao Wang, Zhenghao Song, Dingling Zhang, et al. Omnivideobench: Towards audio-visual understanding evaluation for omni mllms. arXiv preprint arXiv:2510.10689, 2025. Li et al. [2026] Xiaoxi Li, Wenxiang Jiao, Jiarui Jin, Haoxuan Li, Hao Wang, Shijian Wang, Guanting Dong, Jiajie Jin, Yinuo Wang, Yuan Lu, et al. Omnigaia: Towards native omni-modal ai agents. arXiv preprint arXiv:2602.22897, 2026. Lin et al. [2025] Yueqian Lin, Yuzhe Fu, Jingyang Zhang, Yudong Liu, Jianyi Zhang, Jingwei Sun, Hai Helen Li, and Yiran Chen. Speechprune: Context-aware token pruning for speech information retrieval. In 2025 IEEE International Conference on Multimedia and Expo (ICME), 2025. Liu et al. [2025] Xuyang Liu, Yiyu Wang, Junpeng Ma, and Linfeng Zhang. Video compression commander: Plug-and-play inference acceleration for video large language models. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025. OpenAI [2026] OpenAI. Gpt-5.5 system card. https://openai.com/index/gpt-5-5-system-card/, 2026. Accessed: 2026-06. Park et al. [2026] Yeo Jeong Park, Hyemi Jang, Minseo Choi, Jongsun Lee, Jooyoung Choi, and Yongkweon Jeon. Omnidrop: Layer-wise token pruning for omni-modal llms via query-guidance. arXiv preprint arXiv:2605.14458, 2026. Rodriguez Alex et al. [2014] A Rodriguez Alex, Alessandro Laio, et al. Clustering by fast search-and-find of density peaks. science, 2014. Shen et al. [2026] Leqi Shen, Guoqiang Gong, Tao He, Yifeng Zhang, Pengzhang Liu, Sicheng Zhao, et al. Fastvid: Dynamic density pruning for fast video large language models. Advances in Neural Information Processing Systems, 2026. Shi et al. [2025] Yang Shi, Jiaheng Liu, Yushuo Guan, Zhenhua Wu, Yuanxing Zhang, Zihao Wang, Weihong Lin, Jingyun Hua, Zekun Wang, Xinlong Chen, et al. Mavors: Multi-granularity video representation for multimodal large language model. In Proceedings of the 33rd ACM International Conference on Multimedia, 2025. Shu et al. [2025] Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. Audio-visual llm for video understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2025. Tan et al. [2025] Xudong Tan, Peng Ye, Chongjun Tu, Jianjian Cao, Yaoxin Yang, Lin Zhang, Dongzhan Zhou, and Tao Chen. Tokencarve: Information-preserving visual token compression in multimodal large language models. arXiv preprint arXiv:2503.10501, 2025. Tang et al. [2025] Changli Tang, Yixuan Li, Yudong Yang, Jimin Zhuang, Guangzhi Sun, Wei Li, Zejun Ma, and Chao Zhang. video-salmonn 2: Caption-enhanced audio-visual large language models. arXiv preprint arXiv:2506.15220, 2025. Tao et al. [2025] Keda Tao, Can Qin, Haoxuan You, Yang Sui, and Huan Wang. Dycoke: Dynamic compression of tokens for fast video large language models. In Proceedings of the Computer Vision and Pattern Recognition Conference, 2025. Tao et al. [2026a] Keda Tao, Kele Shao, Bohan Yu, Weiqiang Wang, Jian Liu, and Huan Wang. Omnizip: Audio-guided dynamic token compression for fast omnimodal large language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 17682–17692, 2026a. Tao et al. [2026b] Keda Tao, Yuhua Zheng, Jia Xu, Wenjie Du, Kele Shao, Hesong Wang, Xueyi Chen, Xin Jin, Junhan Zhu, Bohan Yu, et al. Lvomnibench: Pioneering long audio-video understanding evaluation for omnimodal llms. arXiv preprint arXiv:2603.19217, 2026b. Team [2026] Qwen Team. Qwen3. 5-omni technical report. arXiv preprint arXiv:2604.15804, 2026. Wang et al. [2025a] Qixun Wang, Yang Shi, Yifei Wang, Yuanxing Zhang, Pengfei Wan, Kun Gai, Xianghua Ying, and Yisen Wang. Monet: Reasoning in latent visual space beyond images and language. arXiv preprint arXiv:2511.21395, 2025a. Wang et al. [2026] Qixun Wang, Yang Shi, Letian Cheng, Zhuoran Zhang, Yan He, Yuqi Tang, Qi Zhang, Xinlei Yu, Ruizhe Chen, Tianrun Xu, Yuanxing Zhang, Pengfei Wan, Haotian Wang, and Xianghua Ying. Beacon: Knowing when and how to perform agentic visual reasoning, 2026. Wang et al. [2025b] Weiyun Wang, Zhangwei Gao, Lixin Gu, Hengjun Pu, Long Cui, Xingguang Wei, Zhaoyang Liu, Linglin Jing, Shenglong Ye, Jie Shao, et al. Internvl3. 5: Advancing open-source multimodal models in versatility, reasoning, and efficiency. arXiv preprint arXiv:2508.18265, 2025b. Wu et al. [2025] Peng Wu, Wanshun Su, Xiangteng He, Peng Wang, and Yanning Zhang. Varcmp: Adapting cross-modal pre-training models for video anomaly retrieval. In Proceedings of the AAAI Conference on Artificial Intelligence, 2025. Wu et al. [2026] Peng Wu, Wanshun Su, Guansong Pang, Yujia Sun, Qingsen Yan, Peng Wang, and Yanning Zhang. Avadclip: Audio-visual collaboration for robust video anomaly detection. IEEE Transactions on Multimedia, 2026. Xin et al. [2026] Zijie Xin, Jie Yang, Ruixiang Zhao, Tianyi Wang, Fengyun Rao, Jing Lyu, and Xirong Li. Stage-adaptive token selection for efficient omni-modal llms. arXiv preprint arXiv:2605.20035, 2026. Xing et al. [2024] Long Xing, Qidong Huang, Xiaoyi Dong, Jiajie Lu, Pan Zhang, Yuhang Zang, Yuhang Cao, Conghui He, Jiaqi Wang, Feng Wu, et al. Pyramiddrop: Accelerating your large vision-language models via pyramid visual redundancy reduction. arXiv preprint arXiv:2410.17247, 2024. Xu et al. [2025a] Jin Xu, Zhifang Guo, Jinzheng He, Hangrui Hu, Ting He, Shuai Bai, Keqin Chen, Jialin Wang, Yang Fan, Kai Dang, Bin Zhang, Xiong Wang, Yunfei Chu, and Junyang Lin. Qwen2.5-omni technical report. arXiv preprint arXiv:2503.20215, 2025a. Xu et al. [2025b] Jin Xu, Zhifang Guo, Hangrui Hu, Yunfei Chu, Xiong Wang, Jinzheng He, Yuxuan Wang, Xian Shi, Ting He, Xinfa Zhu, et al. Qwen3-omni technical report. arXiv preprint arXiv:2509.17765, 2025b. Yang et al. [2025a] Qize Yang, Shimin Yao, Weixuan Chen, Shenghao Fu, Detao Bai, Jiaxing Zhao, Boyuan Sun, Bowen Yin, Xihan Wei, and Jingren Zhou. Humanomniv2: From understanding to omni-modal reasoning with context. arXiv preprint arXiv:2506.21277, 2025a. Yang et al. [2025b] Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025b. Yang et al. [2025c] Yudong Yang, Jimin Zhuang, Guangzhi Sun, Changli Tang, Yixuan Li, Peihan Li, Yifan Jiang, Wei Li, Zejun Ma, and Chao Zhang. Audio-centric video understanding benchmark without text shortcut. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, 2025c. Yao et al. [2024] Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800, 2024. Zhang et al. [2025a] Kaichen Zhang, Bo Li, Peiyuan Zhang, Fanyi Pu, Joshua Adrian Cahyono, Kairui Hu, Shuai Liu, Yuanhan Zhang, Jingkang Yang, Chunyuan Li, et al. Lmms-eval: Reality check on the evaluation of large multimodal models. In Findings of the Association for Computational Linguistics: NAACL 2025, 2025a. Zhang et al. [2025b] YiFan Zhang, Yang Shi, Weichen Yu, Qingsong Wen, Xue Wang, Wenjing Yang, Zhang Zhang, Liang Wang, and Rong Jin. Debiasing multimodal large language models via penalization of language priors. In Proceedings of the 33rd ACM International Conference on Multimedia, 2025b. Zhou et al. [2025] Ziwei Zhou, Rui Wang, Zuxuan Wu, and Yu-Gang Jiang. Daily-omni: Towards audio-visual reasoning with temporal alignment across modalities. arXiv preprint arXiv:2505.17862, 2025. Appendix A Benchmark Details 媒体内容 · 前往原文查看 Benchmark #Videos #QA Pairs Duration(sec) AVUT [48]# 691 1,734 69.1 WorldSense [17] 1,662 3,172 140.7 DailyOmni [52] 684 1,197 43.2 VideoMME [12] 900 2,700 1017.9 LVOmniBench [35] 275 1,014 2069.7 # The full AVUT benchmark contains 2,662 videos and 11,609 QA pairs. Following the evaluation setting of OmniZip [34], we use only the human-annotated subset, which contains 691 videos and 1,734 QA pairs. Table 8: Statistics of the five benchmarks used in our experiments. For each benchmark, we report the number of videos, QA pairs, and the average video duration. Table 8 summarizes the five benchmarks used in our evaluation. Together, they contain 4,212 videos and 9,817 QA pairs, covering diverse real-world scenarios and a wide range of temporal scales. These benchmarks differ substantially in their dependence on audio and visual evidence, enabling a comprehensive assessment of token compression across different durations, modalities, and reasoning requirements. In what follows, we briefly introduce each benchmark. AVUT[48] is an audio-centric video understanding benchmark designed to evaluate both audio-content understanding and audio-visual alignment while minimizing textual shortcuts. Its questions cover tasks such as audio information extraction, audio-event localization, and matching auditory events with their corresponding visual entities or actions. Following OmniZip [34], we evaluate on its human-annotated subset, which contains 691 videos and 1,734 QA pairs, with an average duration of 69.1 seconds. WorldSense[17] comprises 1,662 synchronized audio-visual videos spanning 8 major domains and 67 fine-grained categories, together with 3,172 expert-annotated multiple-choice questions across 26 tasks. Its questions are designed with strong audio-visual coupling, requiring models to jointly integrate evidence from both modalities rather than relying on either modality alone. The videos have an average duration of 140.7 seconds. DailyOmni[52] consists of 684 videos collected from diverse daily-life scenarios and 1,197 multiple-choice QA pairs across 6 major tasks. The benchmark emphasizes temporally aligned audio-visual reasoning, ranging from basic event alignment and temporal ordering to cross-modal inference and contextual reasoning. Its videos contain rich auditory and visual events and have an average duration of 43.2 seconds. VideoMME[12] contains 900 videos covering 6 major domains, with durations ranging from 11 seconds to 1 hour. The videos are divided into short, medium, and long subsets and are accompanied by 2,700 human-annotated multiple-choice QA pairs. VideoMME supports audio inputs, allowing us to evaluate token compression under different modality configurations. Its average video duration is 1,017.9 seconds. LVOmniBench[35] is a long-form audio-visual understanding benchmark containing 275 videos and 1,014 multiple-choice questions. Its videos range from 10 to 90 minutes and have an average duration of 2,069.7 seconds. The benchmark places particular emphasis on long-range audio-visual correspondence, multi-event information retention, and temporal localization, making it well suited for evaluating compression methods under extremely long multimodal contexts. Appendix B More Implementation Details 媒体内容 · 前往原文查看 Pre-LLM Retention Ratio Qwen2.5-Omni-3B/7B MiniCPM-o-2.6 Audio-intact Both-selected Audio-intact Both-selected 25% 12%-100% 20%-55% 11%-100% 20%-50% 20% 7%-100% 15%-50% 5%-100% 15%-45% 15% – 10%-45% – 10%-40% 10% – 6%-35% – 5%-35% Table 9: Model-specific visual-audio allocations for each pre-LLM retention ratio . Each entry gives -, the retained percentages of visual and audio tokens, whose token-count-weighted average approximately matches . Audio-intact keeps all audio tokens, whereas Both-selected compresses both modalities. Retention Ratios and Configuration. The modality-specific retention ratios differ between Qwen2.5-Omni-3B/7B and MiniCPM-o-2.6 because the two model families use different audio and video tokenization schemes. For each temporal window, Qwen2.5-Omni produces approximately 50 audio tokens and 288 visual tokens, whereas MiniCPM-o-2.6 produces approximately 750 audio tokens for every 4,096 visual tokens. We therefore allocate the audio and visual token budgets according to each model’s native tokenization ratio, rather than applying identical modality-specific retention ratios across models, as detailed in Table 9. This preserves a comparable balance between the two modalities under the same overall pre-LLM retention budget. For all models, the maximum pixel budget per sampled frame is set to 1282828. B.1 DPC-KNN Formulation For modality , let denote the input modality tokens and the pre-LLM token budget. After importance selection, the top tokens form . Coverage selection applies DPC-KNN [9, 27] to the remaining candidates using the joint distance: (13) where denotes the temporal distance for audio or spatiotemporal distance for video. We independently normalize the positional distances to within each modality. For each candidate token , let denote its nearest candidates excluding itself under . We set =7 in all experiments. Its local density is defined as: (14) In the following, all indices , , and refer to candidates outside . The separation from higher-density candidates is defined as: (15) The representative score is computed as: (16) A high score indicates that a token is both representative of its local neighborhood and separated from other high-density regions. The top candidates form . Combining them with the importance-selected tokens gives: (17) Appendix C Reproduction Details For each target pre-LLM retention ratio , we use the model-specific visual–audio allocations – in Table 9. Unless otherwise specified, the visual and audio retention ratios of all methods follow these allocations. FastV, VisionZip, FastVID, and VidCom2 are evaluated under the Audio-intact mode, while FastV-om, VisionZip-om, and OmniZip use the Both-selected mode. For the visual-only methods, we keep and reallocate the remaining token budget to visual tokens for each example. FastV[3] (ECCV 2024). FastV prunes visual tokens after the -th LLM block according to the attention scores from the last text token. We follow the official setting with =2. FastV-om applies the same attention-based selection to both visual and audio tokens. At 10%, FastV-om is not reported for Qwen2.5-Omni-7B or MiniCPM-o-2.6 because their target visual retention ratios, 6% and 5%, respectively, are below the minimum layer-averaged retention of 2/287.1% imposed by the first two unpruned blocks. VisionZip[47] (CVPR 2025). VisionZip selects dominant visual tokens using encoder attention and optionally merges redundant tokens into contextual representatives. Since compression at the encoder output conflicts with visual pooling, we apply it after pooling following [28]. We use two temporal grids per visual window and set the contextual-token ratio to , allocating the entire budget to dominant tokens. VisionZip-om applies the same attention-based selection to audio tokens using two-second windows. FastVID[28] (NeurIPS 2025). FastVID dynamically segments video tokens according to adjacent-frame similarities and allocates the visual-token budget based on segment- and frame-level saliency. We apply it to pooled visual tokens and set the minimum number of segments to =8, the segmentation threshold to =0.84, and the anchor-frame step to =4. Within each segment, the token budget of every non-anchor frame is further scaled by 0.7. Since this scaling is applied after budget allocation, the realized retention ratio can be slightly lower than the target . VidCom2[24] (EMNLP 2025). VidCom2 selects informative visual tokens by identifying feature outliers from video- and frame-level representations using multi-scale Gaussian similarities. We apply it to pooled visual tokens with a low-variance channel ratio of 0.5, a temperature of 0.01, and Gaussian scales . Its per-frame retention ratios are dynamically adjusted to match the target visual-token budget. OmniZip[34] (CVPR 2026). OmniZip selects dominant audio tokens using audio-encoder attention and retains an additional 5% contextual representatives, each aggregating at most =3 discarded tokens. The resulting audio retention guides the group-wise video discard ratios, which are normalized to the target visual budget and clipped to [35%,75%]. Within each video group, OmniZip applies DPC-KNN selection to alternating frames and inter-frame similarity-based selection to the remaining frames. We use the official implementation with a contextual ratio of 0.05 and =3. Since the maximum video discard ratio is 75%, cannot be reduced below 25%. Therefore, at R=25%, we use =25%. The audio budget consists of 20% dominant tokens and 5% contextual representatives, both of which are counted toward . OmniSIFT[7] (ICML 2026). OmniSIFT adopts modality-asymmetric token compression, using spatiotemporal pruning to remove intra- and inter-frame visual redundancy and vision-guided selection to filter audio tokens. The original method jointly optimizes these compression modules through alignment training. For a fair comparison with training-free methods, OmniSIFT uses only its token compression strategy without additional training. SEATS[42]. SEATS combines window-wise pre-LLM compression with progressive query-guided compression within the LLM. Before the LLM, it compresses visual and audio tokens within temporal windows. Within the LLM, it progressively removes tokens at predefined layers, followed by optional late-layer token removal. Since late-layer token removal further reduces FLOPs but may also degrade performance, we evaluate two variants for a fair comparison: SEATS follows the original configuration, whereas SEATS disables late-layer removal while retaining both pre-LLM and progressive inner-LLM compression. Both variants are configured to match the final modality-specific retention ratios of our method. SEATS adopts a depth-specific layer-selection schedule whose official configuration is provided only for 28-layer architectures, without an adaptation for the 36-layer Qwen2.5-Omni-3B. To avoid modifying this core strategy, we evaluate both variants only on Qwen2.5-Omni-7B and MiniCPM-o-2.6, which both contain 28 LLM layers. For a fair comparison, we reproduce all methods under a unified evaluation setting and evaluate them using NVIDIA H20 GPUs. Appendix D Computing Cost Evaluation Following prior work [34], we evaluate the computational efficiency of token compression using the floating-point operations (FLOPs) introduced by vision and audio tokens during the prefilling and decoding stages. Let denote the total number of vision and audio tokens, where is the hidden dimension and is the FFN intermediate dimension. Each transformer layer is approximated as a multi-head attention (MHA) module and a feed-forward network (FFN). Under the convention that one multiply–accumulate operation is counted as one FLOP, the prefilling cost per layer is approximated as . During autoregressive decoding, the computational cost of generating tokens is approximated as . We set =100 in all experiments. Therefore, for an LLM with transformer layers and a constant sequence length , the total FLOPs are formulated as: (18) For methods whose sequence length changes across transformer layers, we apply Eq. (18) to each layer using its effective multimodal sequence length and sum the resulting costs over all layers. Pre-LLM compression methods use the compressed sequence throughout the transformer. FastV and FastV-om retain the full multimodal sequence in the first two layers and compress it in subsequent layers. SEATS⋆ progressively reduces tokens after layers 16, 18, and 20, whereas SEATS† additionally discards the remaining multimodal tokens from layer 24 onward. Our method performs its second-stage compression at layer 18 for Qwen2.5-Omni-7B and MiniCPM-o-2.6, and at layer 26 for Qwen2.5-Omni-3B. FastVID additionally reduces non-key-frame tokens by a factor of 0.7. Vision and audio encoders, modality projectors, textual tokens, token-selection operations, and the language-model head are excluded. Figure 4: Visualization of the Pre-LLM and Inner-LLM token compression process. Appendix E More Experimental Details Different Backbones. 媒体内容 · 前往原文查看 Method Efficiency Benchmark Performance Average FLOPs (T) Relative FLOPs Retention Ratio AVUT World Sense Daily Omni MME LVOmni Bench Avg. Score Relative Perf. (%) \rowcolorgray!18 Qwen2.5-Omni-3B 37.4 100.0% 100%/100% 62.5 46.5 61.0 62.5 35.2 53.5 100.0 Random 6.7 18.1% 25%/– 56.3 42.7 54.5 61.1 33.6 49.6 92.7 FastV-om 8.5 22.6% 100%/25% 55.4 43.3 54.2 60.3 32.9 49.2 92.0 OmniZip 6.7 18.1% 25%/– 54.4 42.4 51.8 61.1 32.8 48.5 90.7 OmniSIFT 6.7 18.1% 25%/– 57.0 42.6 50.5 61.8 34.3 49.2 92.0 OmniPack (w/o M) 6.7 18.1% 25%/– 58.3 45.2 55.4 61.7 33.6 50.8 95.0 \rowcolorblue!8 OmniPack 5.8 15.5% 25%/12.5% 57.9 45.1 55.3 61.7 33.8 50.8 95.0 Random 5.3 14.2% 20%/– 55.7 42.4 51.3 60.1 34.6 48.8 91.2 FastV-om 7.1 18.9% 100%/20% 54.1 42.7 51.6 57.6 31.1 47.4 88.6 OmniSIFT 5.3 14.2% 20%/– 54.2 39.9 51.0 61.1 33.3 47.9 89.5 OmniPack (w/o M) 5.3 14.2% 20%/– 58.2 44.4 54.2 61.1 33.7 50.3 94.0 OmniPack 4.6 12.2% 20%/10% 58.2 44.4 53.9 61.0 33.6 50.2 93.8 Random 2.6 7.0% 10%/– 51.7 38.9 49.0 56.8 32.1 45.7 85.4 FastV-om 4.5 12.2% 100%/10% 47.6 40.2 49.4 48.6 29.7 43.1 80.6 VisionZip-om 2.6 7.0% 10%/– 52.6 42.1 50.5 57.7 31.3 46.8 87.5 OmniSIFT 2.6 7.0% 10%/– 49.9 38.2 48.1 56.6 32.2 45.0 84.1 OmniPack (w/o M) 2.6 7.0% 10%/– 55.6 42.8 50.1 58.9 32.8 48.0 89.7 \rowcolorblue!8 OmniPack 2.3 6.1% 10%/5% 55.7 42.5 50.0 58.6 32.7 47.9 89.5 \rowcolorgray!18 MiniCPM-o-2.6 73.2 100.0% 100%/100% 53.2 40.8 52.2 62.0 29.5 47.5 100.0 Random 14.9 20.4% 25%/– 52.8 39.8 53.9 61.5 35.0 48.6 102.3 FastV-om 19.1 26.1% 100%/25% 49.6 37.1 50.4 52.7 30.1 44.0 92.6 VisionZip-om 14.9 20.4% 25%/– 50.9 40.4 53.3 60.9 35.5 48.2 101.5 OmniSIFT 14.9 20.4% 25%/– 50.6 40.4 52.3 60.0 33.2 47.3 99.6 SEATS 11.6 15.8% 25%/12.5% 53.2 40.3 53.4 60.6 34.9 48.5 102.1 SEATS 12.5 17.1% 25%/12.5% 52.9 40.5 53.1 62.1 33.9 48.5 102.1 OmniPack (w/o M) 14.9 20.4% 25%/– 52.9 40.8 54.0 63.5 34.8 49.2 103.6 \rowcolorblue!8 OmniPack 12.2 16.7% 25%/12.5% 53.8 40.9 54.1 63.4 34.2 49.3 103.8 Random 11.8 16.2% 20%/– 51.2 39.2 53.1 61.3 35.0 48.0 101.1 FastV-om 16.2 22.2% 100%/20% 48.0 36.6 48.4 53.6 29.9 43.3 91.2 VisionZip-om 11.8 16.2% 20%/– 49.9 39.7 53.0 61.1 35.2 47.8 100.6 OmniSIFT 11.8 16.2% 20%/– 50.5 39.0 51.6 60.0 33.3 46.9 98.7 SEATS 9.2 12.6% 20%/10% 50.6 40.1 53.0 60.8 36.3 48.2 101.5 SEATS 10.0 13.6% 20%/10% 50.7 40.3 52.5 61.7 32.9 47.6 100.2 OmniPack (w/o M) 11.8 16.2% 20%/– 52.6 40.5 53.6 63.7 35.4 49.2 103.6 \rowcolorblue!8 OmniPack 9.8 13.3% 20%/10% 52.8 40.3 53.5 63.0 35.8 49.1 103.4 Random 6.0 8.2% 10%/– 46.0 37.1 47.0 58.8 32.8 44.3 93.3 VisionZip-om 6.0 8.2% 10%/– 45.5 37.6 47.3 58.1 33.8 44.5 93.7 OmniSIFT 6.0 8.2% 10%/– 45.1 36.5 47.6 57.1 34.2 44.1 92.8 SEATS 4.7 6.5% 10%/5% 46.9 37.6 47.8 57.8 33.6 44.7 94.1 SEATS 5.1 7.0% 10%/5% 47.1 37.5 46.3 58.1 33.6 44.5 93.7 OmniPack (w/o M) 6.0 8.2% 10%/– 50.1 38.7 48.6 60.1 33.8 46.3 97.5 \rowcolorblue!8 OmniPack 5.0 6.8% 10%/5% 47.0 38.7 48.7 61.2 33.6 45.8 96.4 Table 10: Cross-backbone comparison on Qwen2.5-Omni-3B and MiniCPM-o-2.6. Beyond the 15% retention setting reported in the main paper, Table 10 provides additional results at 25%, 20%, and 10% pre-LLM retention ratios on Qwen2.5-Omni-3B and MiniCPM-o-2.6. On Qwen2.5-Omni-3B, OmniPack retains 95.0%, 93.8%, and 89.5% of the original performance while using only 15.5%, 12.2%, and 6.1% of the original FLOPs, respectively. On MiniCPM-o-2.6, OmniPack achieves 103.8% and 103.4% of the original performance at 25% and 20% retention, and retains 96.4% at 10% retention with only 6.8% of the original FLOPs. These results demonstrate a consistent performance–efficiency trade-off across different model scales, architectures, and compression budgets. 媒体内容 · 前往原文查看 Method Efficiency Fine-Grained Performance Avg. FLOPs (T) Relative FLOPs Retention Ratio Audio Event Location Audio Object Matching Audio OCR Matching Audio Information Extraction Audio Character Matching Audio Content Counting \rowcolorgray!18 Qwen2.5-Omni-7B 73.2 100.0% 100% / 100% 35.9 67.2 58.2 85.9 68.1 40.7 64.0 Random 14.9 20.4% 25% / – 32.9 53.3 61.4 82.8 59.5 36.4 58.7 FastV-om 19.1 26.1% 100% / 25% 38.2 52.3 68.8 83.4 55.2 35.6 59.3 VisionZip-om 14.9 20.4% 25% / – 36.5 52.0 63.7 82.2 59.2 33.1 58.7 OmniZip 14.9 20.4% 25% / – 37.0 53.8 64.3 70.8 59.7 27.9 56.9 OmniSIFT 14.9 20.4% 25% / – 37.6 51.0 67.2 80.4 56.8 33.9 58.4 SEATS 11.6 15.8% 25% / 12.5% 36.5 54.6 66.6 84.0 58.0 35.6 60.0 SEATS 12.5 17.1% 25% / 12.5% 35.9 54.9 66.6 84.0 58.3 35.6 60.1 OmniPack (w/o M) 14.9 20.4% 25% / – 39.4 53.3 66.2 83.4 61.4 36.4 60.7 \rowcolorblue!8 OmniPack 12.2 16.7% 25% / 12.5% 39.4 53.6 67.5 83.4 61.6 35.6 61.0 Table 11: Fine-grained performance comparison on six AVUT subtasks using Qwen2.5-Omni-7B. 媒体内容 · 前往原文查看 Method Efficiency Fine-Grained Performance Avg. Prefill FLOPs (T) Relative FLOPs Retention Ratio Tech & Science Culture & Politics Daily Life Film & TV Performance Games Sports Music \rowcolorgray!18 Qwen2.5-Omni-7B 73.2 100.0% 100% / 100% 52.4 51.1 48.3 44.1 42.0 41.0 41.7 47.3 46.6 Random 14.9 20.4% 25% / – 48.8 44.7 44.5 41.7 38.2 36.9 39.1 45.1 43.1 FastV 19.1 26.1% 100%/25% 47.8 44.7 42.4 39.6 39.3 36.5 37.9 44.1 42.0 FastV-om 19.1 26.1% 100%/25% 49.6 46.3 43.8 40.9 39.0 37.3 39.8 44.8 43.3 VisionZip 14.9 20.4% 25% / – 49.6 48.2 44.2 40.4 42.7 40.8 38.8 45.6 44.0 VisionZip-om 14.9 20.4% 25% / – 48.4 50.5 46.2 39.8 40.1 41.6 40.5 47.5 44.7 FastVID 13.6 18.5% 25% / – 47.4 45.3 43.0 40.6 40.8 37.8 39.3 45.1 42.8 VidCom2 14.9 20.4% 25% / – 48.2 47.3 45.0 39.8 39.3 37.8 40.0 44.1 43.3 OmniZip 14.9 20.4% 25% / – 47.8 44.0 43.6 38.5 37.5 41.2 38.4 46.3 42.6 OmniSIFT 14.9 20.4% 25% / – 45.9 46.6 44.5 37.5 37.8 36.9 38.6 43.6 42.1 SEATS 11.6 15.8% 25%/12.5% 48.2 51.1 46.8 40.6 40.8 39.5 41.6 46.8 45.0 SEATS 12.5 17.1% 25%/12.5% 48.4 51.1 46.8 41.4 40.8 39.1 41.2 46.8 45.0 OmniPack (w/o M) 14.9 20.4% 25% / – 49.8 49.8 46.5 43.0 43.0 38.2 39.8 48.3 45.3 \rowcolorblue!8 OmniPack 12.2 16.7% 25%/12.5% 49.4 50.2 45.4 43.0 42.7 39.5 40.0 47.5 45.3 Table 12: Fine-grained performance comparison on eight WorldSense categories using Qwen2.5-Omni-7B. 媒体内容 · 前往原文查看 Method Efficiency Fine-Grained Performance Avg. FLOPs (T) Relative FLOPs Retention Ratio Event Sequence AV Event Alignment Inference Reasoning Context Understanding Comparative \rowcolorgray!18 Qwen2.5-Omni-7B 73.2 100.0% 100% / 100% 58.2 52.5 76.6 78.9 59.1 70.2 63.9 Random 14.9 20.4% 25% / – 49.4 41.6 73.4 73.7 52.3 69.5 57.1 FastV-om 19.1 26.1% 100%/25% 50.3 44.5 78.6 77.1 52.3 64.9 58.7 VisionZip-om 14.9 20.4% 25% / – 50.3 45.8 77.3 74.3 49.7 67.2 58.2 OmniZip 14.9 20.4% 25% / – 50.7 41.6 71.4 72.0 47.7 66.4 55.9 OmniSIFT 14.9 20.4% 25% / – 49.7 43.3 77.3 72.6 52.3 70.2 58.0 SEATS 11.6 15.8% 25%/12.5% 53.9 46.2 76.6 75.4 51.8 67.9 59.7 SEATS 12.5 17.1% 25%/12.5% 52.9 47.1 77.9 76.6 51.3 67.9 59.8 OmniPack (w/o M) 14.9 20.4% 25% / – 53.9 47.5 79.8 74.3 52.3 68.7 60.3 \rowcolorblue!8 OmniPack 12.2 16.7% 25%/12.5% 53.6 48.7 80.5 74.3 52.9 67.9 60.6 Table 13: Fine-grained performance comparison on six DailyOmni subtasks using Qwen2.5-Omni-7B. 媒体内容 · 前往原文查看 Method Efficiency Fine-Grained Performance Avg. FLOPs (T) Retention Ratio Short Medium Long \rowcolorgray!18 Qwen2.5-Omni-7B 73.2 100% / 100% 77.1 63.4 52.8 64.4 Random 8.9 15% / – 69.8 63.8 55.4 63.0 FastV-om 13.5 100% / 15% 62.2 59.1 51.2 57.5 VisionZip-om 8.9 15% / – 68.3 62.9 53.6 61.6 OmniSIFT 8.9 15% / – 69.7 65.1 53.8 62.9 SEATS 6.9 15% / 7.5% 71.6 65.1 54.2 63.6 SEATS 7.5 15% / 7.5% 71.8 65.1 54.4 63.8 OmniPack (w/o M) 8.9 15% / – 72.4 65.2 56.4 64.7 \rowcolorblue!8 OmniPack 7.3 15% / 7.5% 72.6 65.1 56.7 64.7 Table 14: Duration-wise performance comparison on VideoMME using Qwen2.5-Omni-7B . Fine-Grained Results. Tables 11, 12, 13, and 14 provide fine-grained comparisons across AVUT, WorldSense, DailyOmni, and VideoMME. At the 25%/12.5% retention setting, OmniPack performs particularly well in audio-event localization and character matching on AVUT, as well as audio-visual event alignment, inference, and context understanding on DailyOmni. These tasks require associating acoustic evidence with relevant visual content, demonstrating that our audio-visual collaboration preserves complementary cross-modal cues under aggressive compression. The consistent performance across diverse WorldSense categories further shows that this benefit generalizes across different video domains. On VideoMME, OmniPack performs best on both short- and long-duration videos and remains competitive on medium-duration videos at the 15%/7.5% retention setting. In particular, it retains 107.4% of the original long-video performance, indicating that removing redundant audio-visual tokens is especially beneficial for long sequences. Compression Strategies Collaboration. As illustrated in Figure 4, pre-LLM compression removes modality-specific redundancy, while inner-LLM compression refines tokens with task-conditioned representations. Their contributions vary across temporal groups, with audio varying more due to sparse acoustic events and video remaining stable because of continuous visual content. This complementary behavior enables early compression followed by semantic refinement after audio-visual interaction. Case Study. Figure 5 compares SEATS and OmniPack under the same 25% pre-LLM retention ratio. SEATS may discard critical evidence while retaining redundant tokens, resulting in an incorrect prediction. In contrast, OmniPack combines structure-aware selection with delayed query-conditioned audio-visual compression to better preserve task-relevant evidence and remove redundancy. This produces a more compact and informative representation, enabling the correct prediction under aggressive compression. Figure 5: Qualitative comparison at the 25% pre-LLM retention ratio. (a) SEATS misses key tokens and answers incorrectly. (b) OmniPack preserves them and answers correctly. Figure 6: Sensitivity analysis of the importance ratios . Parameter Ablation. As shown in Figure 6 and Table 15, we analyze the sensitivity of the importance ratios , positional-distance weight , merging thresholds , and merging weight . The importance-ratio analysis shows that =(0.25,0.35) provides a favorable balance between the two modalities. Across the three benchmarks, varying results in at most a 1.0-point difference, while the variations caused by and remain within 0.5 and 0.2 points, respectively. Moreover, all tested configurations consistently outperform existing methods, demonstrating that our gains do not rely on carefully tuned hyperparameters. We therefore adopt =0.20, =(0.10,0.05), and =0.10 as a balanced configuration across different benchmarks. 媒体内容 · 前往原文查看 Parameter Value AVUT World Sense Daily Omni 0.10 58.0 45.1 58.2 0.20 58.1 44.6 58.2 0.30 58.0 44.8 58.2 0.40 58.1 44.5 57.7 0.50 58.5 44.1 57.6 (0.05,0.025) 58.2 44.6 57.7 (0.10,0.05) 58.1 44.6 58.2 (0.15,0.075) 57.8 44.5 58.1 (0.20,0.10) 58.0 44.7 58.0 (0.25,0.125) 57.8 44.8 58.1 0.05 58.1 44.5 58.3 0.10 58.1 44.6 58.2 0.15 57.9 44.7 58.1 0.20 57.9 44.5 58.1 0.25 57.9 44.6 58.1 Table 15: Ablation of compression parameters. Each parameter is varied independently while the others remain fixed at their default values. Bold values indicate the default configuration.