Waymo has outlined the AI architecture behind more than 200 million fully driverless miles, from sensor fusion to safety validation.

The company published an account describing ten lessons drawn from that mileage, covering sensor design, model architecture, and the internal checks it applies before allowing new capabilities onto public roads. Waymo frames the lessons as evidence that its Driver system is making roads safer in the cities it operates in, a claim based on the company’s own safety data rather than independent third-party verification.

Waymo’s findings address what it calls one of the two most argued questions in autonomous vehicle development: whether cameras alone can achieve full autonomy. The company says its data across 200 million miles shows they cannot, and that safe operation at scale requires combining cameras, lidar, and radar.

Each sensor covers a distinct job in that arrangement. Lidar builds the 3D geometry of the surrounding environment to millimetre precision. Cameras handle semantic reading, such as street signs and traffic light colour. Radar tracks the velocity of surrounding objects and continues to function in conditions that degrade camera performance, including heavy rain, fog or dust, according to Waymo.

The second debated question Waymo addresses is whether autonomous vehicles should rely on pre-built high-definition maps at all. Waymo uses them, treating map data as a background reference rather than a live input, similar in function to memory.

Waymo says this lets the onboard computer put its real-time processing budget toward things that are new or changing, such as a temporary stop sign or an unplanned detour, while the map supplies known road geometry during difficult visibility or complex junctions. The company says an AI-driven mapping system keeps this reference layer updated.

Fewer, larger foundation models replace fragmented modules

Waymo describes its earlier architecture as reliant on separate specialised modules, one for pedestrian detection, another for tracking vehicles, and another for reading traffic signals. That approach became difficult to maintain as the system scaled.

The company has since moved toward a smaller number of high-capacity foundation models trained on large datasets, arguing this lets the data determine what is relevant instead of hand-built rules. Waymo compares this to the scaling behaviour seen in large language model development and says it uses teacher-student model pairs to manage the compute available onboard the vehicle.

Consolidating into fewer models did not mean consolidating everything into one undivided system, according to Waymo. The company draws a distinction between end-to-end neural networks (which take in raw sensor pixels and output steering commands directly) and its own approach, which adds a separate onboard validation layer that checks every trajectory the Driver proposes.

That validation layer applies physics-based limits and traffic law checks to each proposed path, using techniques Waymo describes as incorporating reinforcement learning and generative AI-style reasoning. If a proposed manoeuvre would breach a limit or risk a collision, the validation layer blocks it. Waymo believes this separation between planning and validation is a required feature of scaling the system for driverless operation, which it refers to using the SAE Level 4 designation.

Closed-loop simulation lets rare scenarios play out with consequence

Waymo distinguishes between two forms of simulation it uses to test the Driver against events that are hard to encounter in sufficient numbers on real roads, such as a vehicle cutting across three lanes of a freeway. 

Open-loop simulation replays recorded data without the surrounding traffic responding to what the Driver does, which Waymo compares to watching a video rather than participating in a scene. Closed-loop simulation instead lets other simulated vehicles react to the Driver’s actions, so a swerve or hard brake produces a corresponding response from surrounding traffic.

Waymo says this feedback dynamic is what makes closed-loop simulation suited to reinforcement learning techniques and to surfacing edge cases before the vehicle meets them on the road. The company runs tens of billions of simulated miles against the millions of real miles its fleet covers each week.

A separate system, called the ‘Critic’, was built to review driving behaviour recorded both on the road and in simulation. The Critic runs continuously against the weekly mileage the fleet generates, flagging patterns for human engineers to review rather than requiring manual review of everything.

Critic checks for safety and traffic law compliance alongside softer measures such as how smoothly the vehicle handles a turn or how comfortable a braking event feels. The company says this combination, paired with what it calls calibrated driving data, lets it measure Driver quality across scenarios without the system’s own outputs being the sole judge of its performance.

Vision-language models add slower, high-level reasoning

Waymo trains vision-language models – built on Google’s Gemini – to help the Driver interpret situations that fall outside its standard training data, such as a police officer directing traffic with hand signals at a collision site. The company says these models are strong at high-level reasoning about a scene but too slow for direct real-time vehicle control and lack sufficient spatial precision on their own.

To address that, Waymo’s system splits processing into two tracks it describes as fast and slow. Rapid sensor fusion and intuitive processing handle instantaneous control decisions. The vision-language layer handles slower and more deliberative reasoning about long-tail scenarios and anticipated developments in a scene. Waymo calls the combined system its Waymo Foundation Model.

A continuous data flywheel and the limits of simulation alone

Waymo says its fleet generates a constant stream of data from its weekly mileage, and that automated systems including the Critic – along with feedback from riders and the communities it operates in – identify where the Driver needs refinement. 

The company describes a cycle in which relevant data gets extracted, categorised by automated labelling tools, used to retrain models, and then checked again through simulation and the governance framework before redeployment. Waymo says running this cycle across what it calls exabytes of data lets an L4 system work through the long tail of driving situations it encounters.

The company’s final point is that no volume of simulated or supervised miles substitutes for time spent with no human driver present. Waymo argues that improving a driver-assist system, categorised as SAE Level 2, toward full autonomy is a distinct technical path from building a Level 4 system from the outset, one it says has to be validated on closed courses and then hardened through unsupervised driving.

Waymo puts its own total for that unsupervised driving at more than 200 million fully autonomous miles to date.

Learn more about physical AI during the Physical AI Expo held in Amsterdam, London, and North America.

See also: Generalist AI’s GEN-1.5 robot model learns tasks from one demo

Want to learn more about the IoT from industry leaders? Check out IoT Tech Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including AI & Big Data Expo and the Cyber Security Expo. Click here for more information.

IoT News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.