Deedy@deedydas

2026-04-01 11:20·92天前

AI 摘要

Gaussian splats是新兴的实时3D渲染技术，可在iPhone上实现自由视角的沉浸式场景浏览。该技术用高斯分布编码场景结构与外观，相比NeRFs极大提升渲染速度。当前突破包括单图生成（Apple ML SHARP）、动态场景捕捉（4DV ai）及生成模型填补未拍摄区域。未来将成为Vision Pro等VR设备的核心娱乐格式，并与世界模型结合实现城市级漫游或游戏化交互，但仍需解决创建效率、存储传输及视觉真实感等挑战。

I've been obsessed with the most exciting software tech today that's not AI： Gaussian splats.

It's the next generation of videos where you can move around in the scene. And the whole thing renders in realtime on your iPhone.

I went into a pretty deep rabbit hole on it.. so here's some history.

The initial idea was： can we take pictures from different angles and reconstruct a 3D scene？ Fun fact： one of the seminal papers in the field （"Photo Tourism"） was written by a professor I taught graphics for in college， Noah Snavely！ Problem： objects look different at diff angles， because of light etc

Then we had NeRFs which could figure out lighting. Problem： extremely slow.

Gaussian splatting represented a 3D scene with diffuse blobs （gaussians） that encoded structure and appearance. Now， you could take camera shots or drone shots and make a splat in <5s. Problem： a） still needed many images b） splats were static and didn't have video in them c） unseen parts of video or holes are just black or missing

Still need many images？ Apple's ML SHARP can take one image and give you a splat！

Can't have video？ Companies like 4DV ai who made the video below build special capture techniques which allow dynamic scenes to be put in a splat

Parts of the video just black？ Generative models （a subset of world models） can fill in the missing parts not captured by camera.

What does that leave us with？

The future entertainment format whether it's in VR on a Vision Pro or interacting with immersive video are going to use splats. There's still open problrms： a） how do we create splats more efficiently b） how do we store and stream them more efficiently c） how do we make them visually more realistic （lighting d） instead of being a flying camera， can we move like a video game character in the space and interact with objects

Splats are closely related world models and virtual reality. Cool projects like Seoul World Model take street view images and let you fly through any part of the city. It's only a matter of time before the entire world gets a 3D representation we can move through baked straight into Google Maps. Or you can play control a video game character watching a live sports game.

Deedy@deedydas · X

导出 Markdown