“The ChatGPT moment for physical AI is here – when machines begin to understand, reason, and act in the real world,” Nvidia CEO Jensen Huang said in a statement. “Alpamayo brings reasoning to autonomous vehicles, allowing them to think through rare scenarios, drive safely in complex environments, and explain their driving decisions.” 

At the core of Nvidia’s new family is Alpamayo 1, a 10 billion-parameter chain-of-thought, reason-based vision language action (VLA) model that allows an AV to think more like a human so it can solve complex edge cases — like how to navigate a traffic light outage at a busy intersection — without previous experience. 

“It does this by breaking down problems into steps, reasoning through every possibility, and then selecting the safest path,” Ali Kani, Nvidia’s vice president of automotive, said Monday during a press briefing. 

Or as Huang put it during his keynote on Monday: “Not only does [Alpamayo] take sensor input and activate steering wheel, brakes, and acceleration, it also reasons about what action it’s about to take. It tells you what action it’s going to take, the reasons by which it came about that action. And then, of course, the trajectory.”

Alpamayo 1’s underlying code is available on Hugging Face. Developers can fine-tune Alpamayo into smaller, faster versions for vehicle development, use it to train simpler driving systems, or build tools on top of it like auto-labeling systems that automatically tag video data or evaluators that check if a car made a smart decision. 

“They can also use Cosmos to generate synthetic data and then train and test their Alpamayo-based AV application on the combination of the real and synthetic dataset,” Kani said. Cosmos is Nvidia’s brand of generative world models, AI systems that create a representation of a physical environment so they can make predictions and take actions. 

As part of the Alpamayo rollout, Nvidia is also releasing an open dataset with more than 1,700 hours of driving data collected across a range of geographies and conditions, covering rare and complex real-world scenarios. The company is additionally launching AlpaSim, an open source simulation framework for validating autonomous driving systems. Available on GitHub, AlpaSim is designed to recreate real-world driving conditions, from sensors to traffic, so developers can safely test systems at scale.