函数注意力:从成对亲和性到函数对应
阅读原文· arxiv.orgFunctional Attention 将注意力机制重新解释为自适应基之间的函数对应,受几何函数映射启发,用结构化线性算子替代 softmax 亲和性,从而得到紧凑、可泛化且分辨率不变的表示,显式捕捉全局依赖。实验表明,该方法在求解 PDE、3D 分割和回归等算子学习任务中达到 SOTA 性能,并对不同离散化保持鲁棒。
Learning mappings between infinite-dimensional function spaces, or operator learning, is essential for many machine learning applications. Although transformer-based operators are popular, they often rely on token-wise attention. These methods treat continuous fields as discrete tokens and usually ignore the global functional structure. We introduce Functional Attention, which reinterprets attention as a functional correspondence between adaptive bases. Inspired by geometric functional maps, our method replaces softmax affinities with structured linear operators. This yields a compact, generalizable, resolution-invariant representation that explicitly captures global dependencies. Experiments demonstrate that Functional Attention can match state-of-the-art performance in many operator learning tasks, including solving PDEs, 3D segmentation, and regression, while remaining robust to varying discretizations. Project page is available at https://github.com/xjffff/FUNCATTN.