# RayRoPE：面向多视图注意力机制的投影射线位置编码

- 来源：Apple Machine Learning Research（RSS）
- 发布时间：2026-07-20 08:00
- AIHOT 分数：34
- AIHOT 链接：https://aihot.virxact.com/items/cmrtcbcda2734bitloi2ap07h
- 原文链接：https://machinelearning.apple.com/research/rayrope-projective-ray

## AI 摘要

Apple 与 CMU 联合提出 RayRoPE，一种针对多视图 Transformer 的位置编码方案。它基于关联射线并利用沿射线预测的 3D 点进行几何感知编码，通过计算查询帧投影坐标实现 SE(3) 不变性，并能在预测点不精确时解析计算期望位置编码。在新视图合成与立体深度估计任务中，RayRoPE 在 CO3D 数据集上的 LPIPS 指标相对提升 15%，且可无缝融合 RGB-D 输入。

## 正文

We study positional encodings for multi-view transformers that process tokens from a set of posed input images, and seek a mechanism that encodes patches uniquely, allows SE(3)-invariant attention with multi-frequency similarity, and can be adaptive to the geometry of the underlying scene. We find that prior (absolute or relative) encoding schemes for multi-view attention do not meet the above desiderata, and present RayRoPE to address this gap. RayRoPE represents patch positions based on associated rays but leverages a predicted point along the ray instead of the direction for a geometry-aware encoding. To achieve SE(3) invariance, RayRoPE computes query-frame projective coordinates for computing multi-frequency similarity. Lastly, as the ‘predicted’ 3D point along a ray may not be precise, RayRoPE presents a mechanism to analytically compute the expected position encoding under uncertainty. We validate RayRoPE on the tasks of novel-view synthesis and stereo depth estimation and show that it consistently improves over alternate position encoding schemes (e.g. 15% relative improvement on LPIPS in CO3D). We also show that RayRoPE can seamlessly incorporate RGB-D input, resulting in even larger gains over alternatives that cannot positionally encode this information.

† Carnegie Mellon University

Related readings and updates.

Segmental Attention Decoding with Long Form Acoustic Encodings

We address the fundamental incompatibility of attention-based encoder-decoder (AED) models with long-form acoustic encodings. AED models trained on segmented utterances learn to encode absolute frame positions by exploiting limited acoustic context beyond segment boundaries, but fail to generalize when decoding long-form segments where these cues vanish. The model loses ability to order acoustic encodings due to permutation invariance of keys and…

High Fidelity 3D Reconstructions with Limited Physical Views

November 29, 2021research area Computer Visionconference 3DV

Multi-view triangulation is the gold standard for 3D reconstruction from 2D correspondences, given known calibration and sufficient views. However in practice expensive multi-view setups — involving tens sometimes hundreds of cameras — are required to obtain the high fidelity 3D reconstructions necessary for modern applications. In this work we present a novel approach that leverages recent advances in 2D-3D lifting using neural shape priors…