现实世界广阔无垠，物理 AI 系统要在其中运作，就必须理解场景如何变化、预测接下来可能发生什么，并判断某个动作将如何影响世界。机器在工厂、仓库、医院等边缘环境中运行。要在这些环境中理解并采取行动，它们需要能够在内存受限的系统上提供数据中心级性能的模型。
今天，我们在 Hugging Face 的 Cosmos 3 仓库中发布了 NVIDIA Cosmos 3 Edge。这是一个拥有 40 亿参数的开源世界模型，可帮助机器人和视觉 AI 智能体理解周围环境、实时推理，并在边缘设备上生成机器人动作。
下载模型：https://huggingface.co/nvidia/Cosmos3-Edge
该模型在 NVIDIA 边缘计算机（包括 NVIDIA RTX PRO GPU、NVIDIA DGX、NVIDIA GeForce RTX™ GPU，以及包含新发布的 Jetson T2000 和 T3000 模块在内的 NVIDIA Jetson）上实现了内存高效、高吞吐量的推理。
该模型被设计为一个紧凑型开源模型，可作为小型视觉语言模型（VLM）使用，具备同类最佳的吞吐量和精度，并支持实时推理。
作为经过后训练的世界动作模型（WAM），Cosmos 3 Edge 以机器人控制分辨率（640×360 观测）运行，在 NVIDIA Jetson Thor 上实现实时推理，每次推理生成 32 个动作，同时以 15 Hz 的频率实现实时控制。
在同类规模（40 亿参数）的模型中，Cosmos 3 Edge 在视觉分析领域的 VANTAGE-Bench 基准测试中排名第一，并在机器人策略学习方面达到业界领先水平，为智能基础设施和机器人技术树立了新的标杆。
什么是世界建模？
世界模型学习环境如何随时间变化。它表征物体、运动、空间关系以及动作的效果。
设想一个机器人伸手去抓取物体。识别物体仅仅是第一步。机器人还必须理解物体在哪里、它的夹爪如何移动、接触时可能发生什么，以及哪个动作最有可能成功完成任务。
世界模型帮助机器人推理这些关系。它可以预测某个动作的视觉结果，推断导致变化的动作，或生成一个动作以产生期望的结果。
Cosmos 3 Edge 将这些能力整合到一个模型中，并在设备端运行。其共享表示使物理 AI 系统能够理解当前世界状态，模拟可能的未来，并将这些未来与动作联系起来。
两个 Transformer 塔，一个共享表示
Cosmos 3 结合了两个 Transformer 塔：
- 自回归塔：处理视觉和文本 token，用于理解和推理。
- 扩散塔：处理视觉、音频和动作 token，用于预测、生成和神经模拟。
这两个塔保持独立的归一化层和多层感知机。它们共享多模态注意力层，用于对齐语言、视频、音频和动作之间的信息。
这种设计允许模型在生成输出之前先对场景进行推理。根据任务的不同，Cosmos 3 可以从自回归塔生成推理 token，或从扩散塔生成去噪后的视频和动作 token。
注意力模式也针对每种信息类型进行了调整。语言使用因果注意力，其中每个 token 关注其之前的所有 token。扩散 token 则更广泛地关注可用上下文，以支持连贯的预测和生成。
这些组件共同为模型提供了一个关于“正在发生什么”、“接下来可能发生什么”以及“动作如何影响结果”的通用表示。
用于动作的通用表示
物理系统以不同方式描述动作。车辆可能通过自身位姿和运动来表示动作。摄像头使用摄像头运动。机械臂使用其末端执行器的位姿，而手或夹爪还需要表示抓取状态。
Cosmos 3 将这些不同的具身形态映射到一个通用的动作表示中。
动作被编码为紧凑的几何向量，捕捉以下信息：
- 平移
- 旋转
- 操作状态
这在控制与世界的视觉结构之间建立了直接联系。模型能够将像素的变化与物理运动、空间关系以及控制输入关联起来。
因此，生成的视频不再仅仅是视觉预测。它可以表征世界如何因某个动作而预期发生变化。这为开发者提供了基于运动、控制以及因果关系的训练数据。
策略模式
作为策略，Cosmos 3 会预测一个动作及其预期的视觉结果。
该模型可以生成系统应执行的操作，并模拟接下来可能发生的情况。这将世界建模直接与机器人策略训练和评估联系起来。
输入当前状态，输出动作和视觉结果。
这些模式使一个模型能够学习原因、结果和策略。动作可以在模型中双向流动，使 Cosmos 3 Edge 能够预测某个动作的效果，或从其效果推断出该动作。
提示词：拿起香蕉，把它放进盘子里。
我们还发布了 Cosmos 3 Edge 策略（DROID）：一个在 DROID 数据集上针对拾取和放置任务进行后训练的机器人操作策略，并附带了后训练脚本。
开发者可以使用一个小型 H100 集群或 NVIDIA DGX Station，在将 Cosmos 3 Edge 部署到 NVIDIA 边缘和加速计算平台之前，针对其目标工作负载进行高效的微调。
用于提升性能的后训练样本
除了基础模型，我们还发布了参考后训练检查点和训练方案，以帮助开发者针对自身应用对 Cosmos 3 进行适配和优化。
Cosmos 3 是一个开放世界基础模型平台。随着模型使用高质量、特定领域的数据进行后训练，它们在专业应用中的准确性持续提升。Cosmos 可作为使用开放框架构建领域自适应世界模型的起点。后训练使开发者能够在保持输出质量的同时提升模型性能。为演示这一工作流程，我们还发布了 Cosmos 3 Super 4 步知识蒸馏检查点及一份后训练脚本，为 64B 模型提供更快的参考实现，帮助开发者将 Cosmos 适配到自己的领域，并实现更好的下游性能。
Cosmos 3 Super 4 步（文生图与图生视频）：这些是分布匹配蒸馏检查点和脚本，可将扩散模型的去噪步骤从 35–50 步减少到仅 4 步，在保持图像和视频质量与保真度的同时，实现高达 25 倍的推理加速。
这些示例可作为开发者在此基础上构建的参考实现。使用开放的训练脚本，你可以对 Cosmos 3 Edge 进行后训练以定制机器人策略，或蒸馏 Cosmos 3 Super 以实现针对你应用场景的更快速图像和视频生成。
试用 Cosmos 3 Edge
下载模型：https://huggingface.co/nvidia/Cosmos3-Edge
Cosmos 3 Edge 通过共享的世界表征连接了理解、预测、模拟和行动。开发者可以将其部署在更靠近传感器的设备端。该模型既可用作推理器，也可用作动作生成器。
使用开放的 Cosmos 框架来定制和专业化这些模型。
查看 Cosmos 3：https://huggingface.co/collections/nvidia/cosmos3
未来展望
我们将持续推进 Cosmos 3 在物理 AI 领域的发展，即将在交互式世界生成、驾驶场景模拟和机器人策略方面带来改进。
我们还在投资于更广泛的硬件上的更快推理和更高效的后训练，以减少构建下游模型所需的时间和算力。
这包括使用 vLLM 等开放推理和优化框架来优化 Cosmos 3 检查点，更多优化和开发者工具即将推出。
The real world is vast and to operate in it physical AI systems need to understand how a scene is changing, anticipate what could happen next, and determine how an action will affect the world. Machines operate at the edge in factories, warehouses, hospitals and beyond. To understand and act in these environments, they need models that deliver data center–level performance on memory-constrained systems.
Today we are releasing NVIDIA Cosmos 3 Edge on Hugging Face’s Cosmos 3 repository. It’s a 4-billion-parameter open world model that helps robots and vision AI agents understand their surroundings, reason in real time and generate robot actions on edge devices.
Download the model: https://huggingface.co/nvidia/Cosmos3-Edge
The model delivers memory-efficient, high-throughput inference across NVIDIA edge computers including NVIDIA RTX PRO GPUs, NVIDIA DGX, NVIDIA GeForce RTX™ GPUs, NVIDIA Jetson including the newly announced Jetson T2000 and T3000 modules.
Designed as a compact open model that can serve as a small vision language model (VLM) with best-in-class throughput and accuracy with real-time inference.
As a post-trained world action model (WAM), Cosmos 3 Edge operates at robot-control resolution (640×360 observations), delivering real-time reasoning and generating 32 actions per inference on NVIDIA Jetson Thor - while achieving real-time control at 15 Hz.
Among similar size (4B parameters) models, Cosmos 3 Edge ranks #1 on VANTAGE-Bench for vision analytics and state-of-the-art for robot policy learning, setting the state of the art for smart infrastructure and robotics.
What Is World Modeling?
A world model learns how an environment changes over time. It represents objects, motion, spatial relationships, and the effects of actions.
Consider a robot reaching for an object. Recognizing the object is only the first step. The robot must also understand where the object is, how its gripper is moving, what may happen when contact occurs, and which action is most likely to complete the task successfully.
A world model helps the robot reason about these relationships. It can predict the visual result of an action, infer the action that caused a change, or generate an action to produce a desired outcome.
Cosmos 3 Edge brings these capabilities together in one model, on-device. Its shared representation enables physical AI systems to understand the current world state, simulate possible futures, and connect those futures to actions.
Two Transformer Towers, One Shared Representation
Cosmos 3 combines two transformer towers:
- Autoregressive tower: Processes vision and text tokens for understanding and reasoning.
- Diffusion tower: Processes vision, audio, and action tokens for prediction, generation, and neural simulation.
The two towers maintain separate normalization layers and multilayer perceptrons. They share multimodal attention layers, which align information across language, video, audio, and action.
This design allows the model to reason about a scene before generating an output. Depending on the task, Cosmos 3 can produce reasoning tokens from the autoregressive tower or denoised video and action tokens from the diffusion tower.
The attention pattern is also adapted to each type of information. Language uses causal attention, where each token attends to the tokens that came before it. Diffusion tokens attend more broadly to the available context, supporting coherent prediction and generation.
Together, these components give the model a common representation of what is happening, what may happen next, and how an action could influence the outcome.
A Common Representation for Action
Physical systems describe actions differently. A vehicle may represent an action through ego pose and movement. A camera uses camera motion. A robot arm uses the pose of its end effector, while a hand or gripper also needs to represent grasp state.
Cosmos 3 maps these different embodiments into a common action representation.
Actions are encoded as compact geometric vectors that capture:
- Translation
- Rotation
- Manipulation state
This creates a direct connection between control and the visual structure of the world. The model can associate changes in pixels with physical motion, spatial relationships, and control inputs.
As a result, generated video becomes more than a visual prediction. It can represent how the world is expected to change in response to an action. This gives developers training data grounded in motion, control, and cause and effect.
Policy Mode
As a policy, Cosmos 3 predicts an action together with its expected visual consequence.
The model can generate what the system should do and simulate what is likely to happen next. This connects world modeling directly to robot policy training and evaluation.
Current state in, action and visual consequence out.
These modes enable one model to learn cause, effect, and policy. Action can flow through the model in both directions, allowing Cosmos 3 Edge to predict the effects of an action or infer an action from its effects.
Prompt: Pick up the banana and put it in the plate.
We are also releasing Cosmos 3 Edge Policy (DROID): A robot manipulation policy post-trained on the DROID dataset for pick-and-place tasks, with accompanying post-training scripts.
Developers can use a small cluster of H100 or NVIDIA DGX Station to efficiently fine-tune Cosmos 3 Edge for their target workloads before deploying to NVIDIA edge and accelerated computing platforms.
Post-trained samples for improved performance
Alongside the base models, we're releasing reference post-trained checkpoints and training recipes to help developers adapt and optimize Cosmos 3 for their own applications.
Cosmos 3 is an open world foundation model platform. As models are post-trained with high-quality, domain-specific data, their accuracy continue to improve for specialized applications. Cosmos serves as the starting point for building domain-adapted world models using open frameworks. Post-training enables developers to improve model performance while preserving output quality. To demonstrate this workflow, we're also releasing the Cosmos 3 Super 4-Step Distillation checkpoint along with a post-training script, providing a faster reference implementation for 64B model that helps developers adapt Cosmos to their own domains and achieve better downstream performance.
Cosmos 3 Super 4-Step (Text-to-Image & Image-to-Video): These are distribution-matching distilled checkpoint and scripts that reduce diffusion from 35–50 denoising steps to just 4, delivering up to 25× faster inference while preserving image and video quality and fidelity.
These examples serve as reference implementations that developers can build on. Using the open training scripts, you can post-train Cosmos 3 Edge for custom robot policies or distill Cosmos 3 Super for faster image and video generation tailored to your application.
Try Cosmos 3 Edge
Download the model: https://huggingface.co/nvidia/Cosmos3-Edge
Cosmos 3 Edge connects understanding, prediction, simulation, and action through a shared world representation.Developers can deploy it closer to their sensors on-device. The model can be used as a reasoner or as an action generator.
Use open Cosmos Frameworks to customize and specialize the models.
Checkout Cosmos 3: https://huggingface.co/collections/nvidia/cosmos3
What's ahead
We continue to advance Cosmos 3 for Physical AI, with upcoming improvements in interactive world generation, driving scenario simulation, and robotics policies.
We are also investing in faster inference and more efficient post-training across a broader range of hardware, reducing the time and compute required to build downstream models.
This includes optimizing Cosmos 3 checkpoints with open inference and optimization frameworks such as vLLM, with more optimizations and developer tooling coming soon.