Semantic vision encoders have become a central visual interface for multimodal understanding and semantic conditioning in image generation. However, their final tokens discard fine-grained visual details, leading to poor pixel reconstruction and limiting their use in reconstruction-sensitive tasks such as image generation and editing. In this work, we ask whether understanding, generation, and editing can be modeled in a single visual representation space built from a pretrained semantic ViT. We show that the frozen Transformer blocks of a semantic ViT are not intrinsically unable to preserve visual details. Instead, the original patch parameterization drives the representation toward semantic abstraction, making fine-grained information difficult to recover from the final tokens. Based on this observation, we introduce Patch Reparameterization, which preserves the original semantic pathway while adding a reconstruction-aware patch embedding that provides fine-grained visual information to the same frozen ViT blocks. The resulting unified representation preserves multimodal understanding while enabling high-fidelity image reconstruction and a favorable reconstruction--generation trade-off. We further scale this representation into UniSpace, an 8B Mixture-of-Transformer-Experts model that performs understanding, generation, and editing in the same visual space without a separate VAE pathway. System-level evaluations demonstrate practical text-to-image generation and instruction-based image editing, showing that a reparameterized pretrained ViT can serve as a unified visual interface for scalable multimodal modeling.
UniSpace:统一视觉表示与可扩展多模态建模
AI 导读
UniSpace 提出 Patch Reparameterization,在保留预训练语义 ViT 原有语义通路的同时,新增重建感知的 patch 嵌入,使同一视觉空间兼顾多模态理解与高保真图像重建。
HuggingFace Daily Papers(社区热门论文)
43
AI 编辑部评分,满分 100UniSpace:统一视觉表示与可扩展多模态建模
UniSpace 提出 Patch Reparameterization,在保留预训练语义 ViT 原有语义通路的同时,新增重建感知的 patch 嵌入,使同一视觉空间兼顾多模态理解与高保真图像重建。
原文 · 保持原样,未翻译原文 · 未翻译
来源:HuggingFace Daily Papers(社区热门论文)· arxiv.org