Safety Pool Announces 100,000 Self-Driving Test Scenarios Ready For Download

Testing robocars is challenging. Because they aren’t human level intelligences able to solve problems with advanced reasoning and thought, you end up needing to throw everything that you can think of that’s likely to happen at them, and make sure they perform well. While you can never test them on everything that’s possible or known, if you test on enough you can gain confidence on how well they will handle other new situations. You can measure how frequently something comes along that they can’t handle.

Today the “Safety Pool” organization has announced a library of over 100,000 test scenarios to run in simulation. As a pool, the Safety Pool gathers contributions from members who put in data in order to get back even more useful data from other members. By sharing, members of the pool make themselves and everybody else safer.

I’ve advocated for this approach for over a decade. Everywhere I talk about robocars, people always ask, “What would a car do if it encountered a situation like this...?” The answer is that any situation that’s been seen in scores of millions of miles of real world driving, along with every scenario dreamed up by people asking questions, and any variation of those scenarios should all go into a library of scenarios. Self-driving teams, academics, and armchair roboticists should be able to build and contribute useful scenarios to the library. Every time there’s a car accident or an unusual situation it should go into the library. Then every car from every company can test that it handles that situation well, and if it doesn’t, it can be fixed.

As such, I am pleased that, partly at my instigation, several companies and institutions have joined together to produce this scenario pool. This includes Deepen.AI, a company in which I am an investor. Other partners include the World Economic Forum (the producers of “Davos”) and WMG at the University of Warwick, along with several founding members ranging from Tier-1 suppliers like Denso and Nvidia NVDA -0.7% to government agencies like Transport Canada and Innovate UK to self-driving teams like Motional to consultants like McKinsey to name just a few.

Most of the leading self-driving teams, most famously Waymo, have built large libraries of their own simulation scenarios. So far, they have viewed these as proprietary assets that help them test and improve their cars faster. This can be true, but many also believe that teams should not compete too much on safety. That’s because it’s better for everybody if everybody is safer. One incident, like Uber’s UBER -2.2% fatality, set back the whole industry, not just Uber. As a pool grows, each contributor can get back many times what they put in, getting back quality as good as they put in. It’s a win for all.

Cars tested with the pool will be able to know they can handle everything that human drivers would see in many lifetimes of driving. In particular, they can be tested against the unusual situations that create risk. They can also be tested against variations of those situations. You can take a situation and say, “try it if the other car shows up 1 second earlier or later, or 2 seconds, or 3. Test it if the car is one foot to the left or right. Test it in heavier or lighter traffic. Test it in rain or snow. Every scenario lets us learn if there is anything that trips up the self-driving system and creates an accident.

Waymo recently impressed by building scenarios for every fatal accident that took place in the last few decades on their pilot project turf in Chandler, Arizona. They ran their car through each accident, and found their cars could almost always prevent the accidents from happening, which they announced with deserved pride.

It would be good if every car could do the same test by trading with Waymo their own equally useful tests. This is the goal of the Safety Pool. After this is done, you can get into any member self-driving car and know it knows how to handle just about anything anybody has seen or thought up, which is more than you can say about must humans.

Read/leave comments at this page.

Follow me on Twitter or LinkedIn. Check out my website.

Disclaimer: The author has stock in Deepen.AI (which is a significant player in the Safety Pool) and in GOOG.