Nvidia Releases Four Cosmos 3 Models That Generate Driving and Robot Scenes
They produce camera and lidar data together and predict how a scene changes as a robot moves. Nvidia also published the fine-tuning recipes, so developers can adapt the models to their own vehicle sensors and robots.
On November 5, Nvidia released four Cosmos 3 models built for virtual driving and robotics experiments. They generate the scene around a vehicle, or predict the next frames a robot will see based on how it moves. All four were created by fine-tuning Cosmos 3 Nano, and each has 16 billion parameters. Under the OpenMDW1.1 license terms, they can be used for both commercial and non-commercial purposes.
How 11 camera views get generated at once
Cosmos3-Nano-Transfer-Auto is a model that generates driving training data in advance. It generates video from up to 11 viewpoints, including wide-angle fisheye cameras, keeping them consistent with one another. Beyond color video, it also produces lidar data. Lidar is a sensor that reads the three-dimensional structure of the surroundings using lasers. The approach is to fix the road layout and traffic flow, then vary weather, lighting, and time of day through text descriptions.
Developers can fine-tune the model for their own sensor configuration and driving environment. Once it is adjusted to match camera placement and lidar settings, it can be used to generate data. Nvidia says it built Cosmos3-Nano-Sim-Auto, which generates successive scenes rapidly, on top of such an adjusted model.
Sim-Auto takes a previously generated scene plus new driving control inputs and produces camera video and lidar data in short segments. That makes it possible to run experiments in which the scene keeps changing according to the vehicle’s next action. Reflecting driving inputs in the scene, however, requires separately setting up an environment to run the simulation and tooling to produce the control signals.
Turning a robot’s next move into video
Cosmos3-Nano-Sim-Transfer converts states computed by a physics simulator into photorealistic video. The simulator handles motion and contact, and the model generates the imagery from information such as distance and object boundaries, combined with text instructions. Physics is left to the simulator, while the model produces only the scene that a robot control AI will see.
Image: huggingface.co
Cosmos3-Nano-Sim-Bimanual predicts the next video directly from just the scene and the robot’s actions. It supports the two-armed robots YAM and AgiBot, which can be driven either by a control AI or by a person using a joystick. Starting from a single image, moving the robot or the camera produces matching imagery. Within a single control run, actions that shift the viewpoint and actions that manipulate objects can be alternated.
What developers need to use them
Nvidia published fine-tuning procedures alongside the models. Developers can adapt them to their own setting using their own driving or robot data. Running the Bimanual model requires Linux, an Nvidia GPU, Docker, and the NVIDIA Container Toolkit. The published server configuration was tested on a single GB200.
Image: huggingface.co
Nvidia notes that small errors in a generated frame can carry into the next one and accumulate. The thing to watch going forward is less how photorealistic the video looks and more whether virtual environments built this way can properly support the training and evaluation of control AI. Nvidia itself lists measuring that scope and its limits as remaining work.