Nvidia drops open-weight Cosmos 3 models to fake reality for robots
Training self-driving cars on public streets gets expensive and legally risky, so silicon giants are building synthetic physics sandboxes instead. The latest open release turns pure math and sensory feedback into reactive video games for machines.
Nvidia published four specialized open-weight models under the Cosmos 3 umbrella, each packing 16 billion parameters and fine-tuned from the lightweight Cosmos 3 Nano base. Distributed under the commercial-friendly OpenMDW1.1 license on Hugging Face, these networks focus on sensory training data rather than churning out entertaining clips for social media feeds.
Two models cater strictly to autonomous vehicles. The Cosmos3-Nano-Transfer-Auto handles batch offline rendering, spitting out synchronized multi-view footage across up to 11 camera perspectives—including wide fisheye optics—while simultaneously generating matching LiDAR point clouds. High-level road architecture is dictated by structured World Scenario Maps, while text prompts let engineers instantly swap afternoon sunshine for blinding blizzard conditions.
To support closed-loop testing, Cosmos3-Nano-Sim-Auto delivers real-time streaming simulations via causal distillation. This setup generates short video bursts conditioned on past actions, ensuring that hitting the virtual accelerator alters the visual scenery logically instead of triggering an incoherent fever dream.
The robotic branch handles physical manipulation. Cosmos3-Nano-Sim-Transfer takes basic simulator telemetry like depth maps, object bounding boxes, and segmentation masks, transforming raw physics equations into photorealistic video feeds. Meanwhile, Cosmos3-Nano-Sim-Bimanual models direct robotic arm movements for platforms like YAM and AgiBot, responding frame-by-frame to algorithmic commands or manual human joystick inputs.
To stop iterative simulations from melting into mush over time, the pipeline relies on a four-stage training recipe culminating in self-forcing, where networks learn directly from their own previous rendering errors. Replicating the benchmark bimanual setup out of the box required an enterprise-grade Nvidia GB200 superchip running on Linux inside Docker containers.
Letting autonomous networks train inside a synthesized matrix avoids the legal headache of smashing physical prototypes into highway guardrails. Still, staking commercial vehicle safety on neural hallucination engines assumes simulated physics will never mistake a fatal concrete wall for a patch of digital fog.
Source: Hugging Face
Comments
This is where the magic happens: AI reads your discussion and rewrites the article based on the most interesting comments. Each strong comment adds points to the meter below. Once the meter is full, the article updates live — no page reload needed.