ReViV:从单目第一人称视频中联合重建观察者与场景的4D框架

HuggingFace Daily Papers(社区热门论文)·2026-07-20 08:00·46天前
AI 导读

ReViV提出首个统一框架,从单目RGB视频同时重建观察者(全身运动、手部、视线)与场景(相机轨迹、深度)的4D动态。基于掩码生成式第一人称Transformer的单前馈架构,无需预计算相机轨迹或分离建模,在HoloAssist、HOT3D、ARCTIC、Aria Digital Twin、TACO等基准上实现全身、手部、视线重建与相机追踪的SOTA精度,深度估计也具竞争力。代码与模型已完全开源。

HuggingFace Daily Papers(社区热门论文)
49AI 编辑部评分,满分 100

ReViV:从单目第一人称视频中联合重建观察者与场景的4D框架

2026-07-20 08:00· 46天前
AI 导读

ReViV提出首个统一框架,从单目RGB视频同时重建观察者(全身运动、手部、视线)与场景(相机轨迹、深度)的4D动态。基于掩码生成式第一人称Transformer的单前馈架构,无需预计算相机轨迹或分离建模,在HoloAssist、HOT3D、ARCTIC、Aria Digital Twin、TACO等基准上实现全身、手部、视线重建与相机追踪的SOTA精度,深度估计也具竞争力。代码与模型已完全开源。

Egocentric devices, such as wearable front-facing cameras, provide a unique perspective for capturing the continuous interaction between a human viewer and the surrounding environment. A holistic and efficient multimodal model capable of reconstructing this 4D representation is therefore highly desirable. However, existing approaches often rely on auxiliary inputs such as pre-computed camera trajectories, treat scene perception and human ego-motion modeling as separate problems despite their strong interdependency, and suffer from slow inference time. To address these limitations, we present ReViV, the first unified framework for holistic egocentric 4D reconstruction that extracts both viewer and view dynamics from a single monocular RGB video. We formulate the task as learning the full joint probability distribution over multimodal signals, including RGB video, camera trajectory, gaze direction, full-body motion, hand motion, and depth. Powered by a Masked Generative Egocentric Transformer, ReViV operates within a single feed-forward architecture to simultaneously reconstruct the temporally consistent 4D reconstruction across the viewer and the view with fast inference speed. Extensive experiments on diverse benchmarks, including HoloAssist, HOT3D, ARCTIC, Aria Digital Twin, and TACO, demonstrate that ReViV achieves state-of-the-art accuracy and efficiency across holistic ego-body, hand, and gaze reconstruction, camera tracking, while maintaining highly competitive egocentric depth estimation without relying on heavy task-specific priors. Code and models are fully open-sourced: https://reviv4d.github.io/.

来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org