# Transformer 需要三个投影吗？--对 QKV 变体的系统研究

- 来源：Hacker News 热门（buzzing.cc 中文翻译）
- 作者：Anon84
- 发布时间：2026-06-05 10:35
- AIHOT 分数：51
- AIHOT 链接：https://aihot.virxact.com/items/cmq0br63a04qzsltr9kpljkp6
- 原文链接：https://arxiv.org/abs/2606.04032

## AI 摘要

一项系统研究探讨 Transformer 注意力机制中是否必须使用三个独立的投影（Query、Key、Value）。通过分析多种 QKV 变体结构，论文对「三投影」这一设计选择进行了系统性评估。

## 正文

Computer Science > Machine Learning

Title:Do Transformers Need Three Projections? Systematic Study of QKV Variants

Abstract:Transformers have become the standard solution for various AI tasks, with the query, key, and value (QKV) attention formulation playing a central role. However, the individual contribution of these three projections and the impact of omitting some remain poorly understood. We systematically evaluate three projection sharing constraints: a) Q-K=V (shared key-value), b) Q=K-V (shared query-key), and c) Q=K=V (single projection). The last two variants produce symmetric attention maps; to address this, we also explore asymmetric attention via 2D positional encodings. Through experiments spanning synthetic tasks, vision (MNIST, CIFAR, TinyImageNet, anomaly), and language modeling (300M and 1.2B parameter models on 10B tokens), we discovered that our transformers perform on par or occasionally better than the QKV transformer. In language modeling, Q-K=V projection sharing achieves 50% KV cache reduction with only 3.1% perplexity degradation. Crucially, projection sharing is complementary to head sharing (GQA/MQA): combining Q-K=V with GQA-4 yields 87.5% cache reduction, while Q-K=V + MQA achieves 96.9%, enabling practical on-device inference. We show that Q-K=V preserves quality because keys and values can occupy similar representational spaces and attention operates in a low-rank regime, whereas Q=K-V breaks attention directionality. Our results systematically characterize projection sharing as an underexplored instance of weight tying in attention, with direct, quantifiable inference memory benefits, particularly valuable for edge deployment. The code is publicly available at this https URL

Comments: Accepted at ICML 2026 (PMLR vol. 306). 26 pages, 12 figures, 16 tables. Code: this https URL Subjects: Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL); Performance (cs.PF) ACM classes: I.2.6; I.2.7; I.2.10 Cite as: arXiv:2606.04032 [cs.LG] (or arXiv:2606.04032v2 [cs.LG] for this version) https://doi.org/10.48550/arXiv.2606.04032 Focus to learn more arXiv-issued DOI via DataCite

Submission history

Access Paper:

View PDF

HTML (experimental)

TeX Source

Current browse context:

References & Citations

NASA ADS

Google Scholar

Semantic Scholar

BibTeX formatted citation

Bookmark

Bibliographic and Citation Tools

Code, Data and Media Associated with this Article

Demos

Recommenders and Search Tools

Author

Venue

Institution

Topic

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.