# 混合线性注意力LLM中的巨量激活：注意力层前尖峰与层间平台

- 来源：Rohan Paul (@rohanpaul_ai)
- 发布时间：2026-08-18 02:54
- AIHOT 分数：27
- AIHOT 链接：https://aihot.virxact.com/items/cmsxmgymq04oiroz0grrrtxwl
- 原文链接：https://x.com/rohanpaul_ai/status/2089425654609740202

## AI 摘要

混合Transformer架构LLM用廉价循环层替代多数注意力层，但保留的少数全注意力层对模型影响远超其数量占比。在Qwen3.5、Kimi Linear、Nemotron-H和Zamba2中，模型在昂贵回溯层前反复产生异常巨大的内部数值，移动或增加该层会相应改变这些极端值的分布。

## 正文

Some newer efficient LLMs (using "hybrid Transformer") are cutting back on expensive attention layer of standard Transformer, and this paper shows what happens internally around the attention layers that remain.

A "hybrid Transformer" simply means a model that replaces many standard attention layers with cheaper recurrent-style layers but keeps a few full-attention layers.

That shortcut saves compute.

But this paper finds that the occasional expensive layers may have a much bigger effect on the model than their small number suggests.

Across Qwen3.5, Kimi Linear, Nemotron-H, and Zamba2, the model repeatedly produced unusually huge internal numbers immediately before those expensive look-back layers.

Move 1 of those layers, and the huge values move with it.

Use more of them, and those extreme values start staying high across larger parts of the network.

– arxiv. org/abs/2608.12149

Title: "Massive Activations in Hybrid Linear Attention LLMs: Pre-Attention Spikes and Inter-Spike Plateaus"