OpenBMB@OpenBMB

2026-06-26 22:01·6天前

AI 摘要

清华自然语言处理实验室（THUNLP）与面壁智能OpenBMB发布论文，重新审视混合LLM架构中高效注意力（如SWA、Mamba-2、GDN）的实际作用。研究发现：高效注意力设计对短上下文Loss影响极小，但长上下文LongPPL差异显著；全注意力承担检索功能，限制其感受野会大幅提升LongPPL，而限制高效注意力几乎无影响。大窗口SWA导致模型懒惰，延迟检索能力形成。简单方法——对小窗口SWA混合架构的全注意力层仅用NoPE（SWA-128-NoPE），即可用极小短上下文代价显著提升长上下文性能。论文认为瓶颈在于全注意力的检索能力能否被有效激活。

Hybrid LLMs are everywhere now： full attention is mixed with efficient modules like SWA， Mamba-2， and GDN. But what does efficient attention actually do inside these models？ 🧵

New work from THUNLP Lab & OpenBMB： "Rethinking the Role of Efficient Attention in Hybrid Architectures." Through scaling laws， mechanistic analysis， and design studies， they reach a counter-intuitive conclusion 👇

📄 arXiv： https://arxiv.org/abs/2606.15378 💻 Code： https://github.com/thunlp/rethinking-hybrid-attention

1️⃣Same destination， different speed： Efficient-attention design barely affects short-context Loss - all seven curves nearly overlap. But on long-context metric LongPPL， early-training gaps are large， with large-window SWA worst of all. With enough training， every hybrid converges to the full-attention level.

2️⃣Full attention carries retrieval： Restricting full attention's receptive field at inference spikes LongPPL across all hybrids； restricting efficient attention barely moves it. Even recurrent mixers with in-principle unbounded receptive fields （like GDN） store little long-range info in their states. Layer-wise probing shows the same pattern： retrieval gains concentrate in the full-attention layers.

3️⃣Large-Window Laziness： A large SWA window already covers most useful dependencies， so the model needn't push full attention to retrieve from afar-delaying retrieval-head formation. It's like a student who won't walk to the library when the reference book is already on the desk. Smaller windows force full attention to do the retrieval work， training it faster.

4️⃣A simple design that works： Apply NoPE to just the full-attention layers of a small-window SWA hybrid （SWA-128-NoPE）. It substantially improves long-context performance with negligible short-context cost. Under an effective training budget， the bottleneck for the long-context capability of hybrid models is not how powerful the efficient attention module is-it is whether full attention's retrieval capability can be effectively activated. Furthermore， strengthening full attention itself can bring greater performance improvements. Read the full paper！ 🚀 #AI #THUNLP #OpenBMB #LLM #Attention #LongContext #HybridArchitecture #NLP

OpenBMB@OpenBMB · X

63导出 Markdown

2026-06-26 22:01·6天前

在 X 看原推· x.com

AI 摘要

Hybrid LLMs are everywhere now： full attention is mixed with efficient modules like SWA， Mamba-2， and GDN. But what does efficient attention actually do inside these models？ 🧵

📄 arXiv： https://arxiv.org/abs/2606.15378 💻 Code： https://github.com/thunlp/rethinking-hybrid-attention