# GQLA：面向硬件自适应的大语言模型解码的分组查询潜在注意力

- 来源：HuggingFace Daily Papers（社区热门论文）
- 发布时间：2026-05-14 08:00
- AIHOT 分数：57
- AIHOT 链接：https://aihot.virxact.com/items/cmpb32ltl142mslnza7n7t7ny
- 原文链接：https://arxiv.org/abs/2605.15250

## AI 摘要

本研究提出分组查询潜在注意力（GQLA），对DeepSeek-V2/V3中的多头潜在注意力（MLA）进行最小修改，使其在一套权重上暴露两种等效解码路径：与MLA相同的MQA吸收路径，以及带有每组扩展缓存的GQA路径。运行时可根据硬件自动选择路径，无需重新训练。单一的GQLA权重能同时适配H100（采用MQA吸收）与H20（采用GQA及多令牌预测）的硬件性能上限，并在GQA路径上支持高达8路的零冗余张量并行。通过扩展TransMLA为TransGQLA，可将预训练的GQA模型转换为GQLA模型。在LLaMA-3-8B上的实验表明，其MQA吸收路径将每令牌的KV缓存压缩至GQA基线的28.125%，同时在分组路径上结构性保留了GQA级别的流量效率。

## 正文

Multi-head Latent Attention (MLA), the attention used in DeepSeek-V2/V3, jointly compresses keys and values into a low-rank latent and matches the H100 roofline almost perfectly. Its trained weights, however, expose only one decoding path - an absorbed MQA form - which ties efficient inference to H100-class compute-bandwidth ratios, forfeits tensor parallelism along the head axis, and yields no Multi-Token Prediction (MTP) gain on commodity inference GPUs such as the export-restricted H20. We propose Group-Query Latent Attention (GQLA), a minimal modification of MLA whose trained weights expose two algebraically equivalent decoding paths over the same parameters: an MQA-absorb path identical to MLA's, and a GQA path with a per-group expanded cache. The runtime picks the path that matches the target hardware - no retraining, no custom kernels - so a single set of GQLA weights pins the rooflines of both H100 (MQA-absorb, s_q=1) and H20 (GQA + MTP, s_q=2), while supporting up to 8-way zero-redundancy tensor parallelism on the GQA path. To avoid pretraining from scratch we extend TransMLA into TransGQLA, which converts a pretrained GQA checkpoint into a GQLA model; on LLaMA-3-8B it compresses the per-token KV cache to 28.125% of the GQA baseline on the MQA-absorb path while structurally preserving GQA-level traffic on the per-group path.