Random Attention 提出随机驱逐 KV cache 以提升长推理效率

HuggingFace Daily Papers(社区热门论文)·2026-09-03 08:00·1天前
AI 导读

Salesforce AI Research 提出 Random Attention,不对缓存 token 打分,而是保留提示词并在每个注意力头内均匀随机驱逐,在四个模型、六个推理任务上匹配最强先验驱逐方法,vLLM 部署下吞吐量高出 32-43%。

HuggingFace Daily Papers(社区热门论文)
52AI 编辑部评分,满分 100

Random Attention 提出随机驱逐 KV cache 以提升长推理效率

2026-09-03 08:00· 1天前
AI 导读

Salesforce AI Research 提出 Random Attention,不对缓存 token 打分,而是保留提示词并在每个注意力头内均匀随机驱逐,在四个模型、六个推理任务上匹配最强先验驱逐方法,vLLM 部署下吞吐量高出 32-43%。

Large language models achieve superior performance on tasks that require extended reasoning, but long chains of thought make the KV cache a severe memory bottleneck. Existing KV cache compression methods share one paradigm: score each cached token by some estimate of how much it will matter later, and keep the top-scoring ones. We show that the selection signal contributes almost nothing. Random Attention keeps the prompt and evicts uniformly at random within each attention head, computing no score at all; across four models and six reasoning tasks it matches the strongest prior evictor while serving 32-43% higher throughput than it in vLLM deployment. Controlled experiments explain this by showing that 1) the prompt is the fragile part of the cache, and most of the gap between selectors is just whether their selection signal happened to keep it; 2) the reasoning trace protects itself against eviction with redundancy at two levels, in the text (the model restates what it still needs as it works) and across attention heads (each keeps its own copy of the trace), so once the prompt is safe, a random draw retains enough copies of what the model still needs, and no score is required to pick them. Our code is publicly available at https://github.com/SalesforceAIResearch/Random-Attention.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org