IT Safety and Reliability Professor Matt Thatcher

2 Agenda for Today l Brief review of the Case of the Killer Robot l Overview of the: –Therac-25 accidents –Denver International Airport Baggage System l Discussion of “How Good Is Good Enough?” –what are our social responsibilities?

3 Killer Robot Summary l The general problems –simple programming error –inadequate safety engineering and testing –poor HCI design –lax culture of safety in Silicon Techtronics l What would change in you replaced one of the characters with an “ethical” person? –would any of these problems have been solved?

4 Matt’s Humble Opinions l Source of the problems –economic incentives »time pressures u exclusive focus on meeting unrealistic deadlines u there was no payoff to the dvmt team based on usability or safety measures u valuing stock price over operator safety »cut corners  keep your job, challenge decisions  get fired –company culture »poor communication all along the company hierarchy »lots of unproductive, unresolved, and unaddressed conflict »inability to consider alternatives »choice of waterfall model instead of prototyping model as development methodology –inexperience and lack of critical skills in key positions »Johnson (hrdwr guy), Reynolds (data processing guy), Samuels (no exp in physics) l Who is most responsible for: –setting appropriate economic incentives –creating an appropriate culture –putting people with the rights skills into the right jobs

5 Therac-25 Accidents (Basis of the Killer Robot Case) l What was Therac-25? –released in 1983 –computerized radiation therapy machine used to treat cancer patients l Who developed it? –Atomic Energy of Canada, Ltd and GCR (French-based company) l What were the key advances of it over its predecessors (Therac-6 and Therac-20)? –move to more complete software-based control »faster set-up »safety checks were now controlled by software (instead of mechanical interlocks)

6 Therac-25 Accidents (What Happened?) l Massively overdosed patients at least 6 times (3 died, 3 seriously disabled) –June 1985 »Marietta, Ga (Linda Knight, 61) –July 1985 »Hamilton, Ont (Donna Gartner, 40) –December 1985 »Yakima, Wash (Janis Tilman) –March 1986 »Tyler, Tx (Isaac Dahl, 33) –April 1986 »Tyler, Tx (Daniel McCarthy) –January 1987 »Yakima, Wash (Anders Engman)

7 Therac-25 Accidents (Example of Contributing UI Problems) l The technician got the patient set up on the table, and went down the hall to start the treatment. She sat down at the terminal: »hit “x” to start the process u she realized she made a mistake, since she needed to treat the patient with the electron beam, not the X-ray beam »hit the “Up” arrow, »selected the “Edit” command, »hit “e” for electron beam, and »hit “enter” (signifying she was ready to start treatment) »the system showed a “beam ready” prompt »she hit “b” to turn the beam therapy on »the system gave her an error message (Malfunction 54) »she overrode the error message l It turns out that the UI showed that it was in electron mode but it was actually in a “hybrid” mode  delivered more than 125 times the normal dose to the patient

8 Therac-25 Accidents (What Were the Problems?) l The Problems –simple programming errors –inadequate safety engineering »ignored the software risks (almost no unit or integration testing at all) »operators were told it was impossible to overdose a patient –poor HCI design –lax culture of safety in the manufacturing co. –problems were not reported quickly to manufacturer or FDA »prompted a 1990 federal law

9 Friendly-Fire Tragedy (Afghanistan)

10 Denver International Airport’s Baggage System

11 Other System Failures l Problems for individuals –billing inaccuracies (electricity, auto insurance) –database inaccuracies (sex crimes DB, 2000 Florida election) l System failures –AT&T (1990) –Galaxy 4 satellite (1998) –London Stock Exchange (2000) –Warehouse Manager –New York School District

12 Increasing Reliability and Safety l Overconfidence l Software reuse l Professional techniques l UI an human factors l Redundancy and self-checking l Warranties l Regulation of safety applications l Self-regulation l Take responsibility!!!

13 Critical Observation l If it is true that: –information technology affects society AND –some choices in computing design are not completely constrained by mathematics, physics, chemistry, etc. THEN –designers, implementers, teachers, and managers of technology make choices that affect society

14 Principle Actors in Software Design l Software Provider –person or organization that creates the software l Software Buyer –person or organization responsible for obtaining the software for its intended use l Software User –person actually using the software l Society –people, other than providers, buyer, or users who can be affected by the software

15 Obligations of the Software Provider l The Provider (itself) –profit and good reputation l The Buyer –help buyer make an informed decision –set testing goals and meet them –provide warnings about untested areas of the software –inform user about testing results and shortcomings –provide a reasonable warranty on functionality and safety l The User –education/training about use and limitations of software –provide technical support and maintenance –provide reasonable protections –provide clear instructions and user manuals l Biggest Responsibility –answer the question “How Good is Good Enough”

16 Obligations of the Software Buyer l The Provider –respect copyrights and don’t steal –pay a fair price for the product –use the product for the correct purpose –don’t blame the provider for incorrect use –make users available for observations, testing, and evaluation l The Buyer (itself) –learn limitations of the software –perform systems acceptance testing (and audit testing) l The User –ensure proper education/training is provided –ensure proper technical support and maintenance is provided –represent the user during development (be a communication link between user and developer) –make sure users are included in the design process –provide reasonable protections (and a safe working environment) l Biggest Responsibility –make sure software is used for intended purpose (think about biometrics)

17 Obligations of the Software User l The Provider –respect copyrights and don’t steal –read documentation l The Buyer –communicate problems and provide feedback (about training, UI, functionality, etc.) –ensure appropriate use of software –make a good faith effort to learn the software and its limitations l The User (herself) –help in the training process l Biggest Responsibility –ensure that the software continues to perform as intended and report problems

18 Final Observation l Software is treated like other commercials products l Software differs in many critical respects from many other products –serious software errors can remain after rigorous testing because of the logical complexity of software –it is difficult to construct uniform software standards that can be subjected to regulation and inspection –software affects an increasingly large number of people due to the proliferation and flexibility of computers –any group can provide software since set-up costs are low

19 Summary l Computer professionals must stop thinking of themselves as technicians l Just like medical doctors, computer professionals are agents of change in people’s lives l Computer professionals have some ethical responsibility for the changes they are creating in society l But this responsibility is complicated because our position as workers in industry diffuses accountability l We must recognize that part of our job includes larger issues of social responsibility