1. Safety and Automated Driving Systems Kyle Vogt, Cruise, October 28, 2015

2. The issue at hand: How do you know a vehicle is safe enough? 2

3. What exists? The Law FMVSS 3 NHTSA / DMV Turns out, it doesn’t tell us

4. What exists? Standards 4 Industry Standards Best Practices

5. Digging deeper… Combine everything that exists  Meet it or Beat it 5 AVs Problem is you could follow what exists, and still end up with an unsafe product because ADS’s add complexity that the existing base does not address. We have to do more.

6. c Where are some of the holes? 6 c Hazard Analysis Methods Lack of specificity Behavioral Requirements

7. Why are traditional hazard analysis techniques not enough? Bottom up approach misses system-level problems – component interactions Rely on chain of event causation model Not good for software 1 2 3

8. 8 How to address those problems Hazards Design and implementation Safety STPA

9. Testing and Validation 9 What’s challenging about ADSs? Complexity How do you build safe complex systems? Solid process and simulation

10. The path to driverless 10 Naturalistic testing is extremely useful, and key for validation, but cannot accomplish necessary coverage To get on the road, the only legitimate way to know a system is safe will be through extensive simulation

11. Thank you. 11

12. Appendix A: Example of weakness of traditional hazard analysis Bhopal accident December 1984, pesticide plant gas leak in Bhopal, India, exposing over 500,000 people to toxic gas None of the safety devices worked Seems incredibly unlikely that all safety systems would fail at once, but in reality, they were not independent failures Traditional methods only consider limited set of causes and miss the more systemic causes (e.g. financial pressure, poor hiring, failure to heed prior warnings) 12

13. Appendix B: Reactor example Chemical reactor: Computer controls (1) flow of catalyst into reactor and (2) flow of water into reflux condenser to cool off reaction Sensors warn of problems If fault detected, computer programmed to leave variables as is, and simply sound alarm 13

14. Extra - actions taken from STPA results Comprehensive analysis of specific social human driving behaviors Crucial point is requirements, have to have top notch requirements that are derived from safety constraints Create behavioral situations that you can test against, and Test them, extensive simulation Real world experience 14