1 Can We Trust the Computer? What Can Go Wrong? Case Study: The Therac-25 Increasing Reliability and Safety Perspectives on Failures, Dependence, Risk, and Progress Computer Models

2 What Can Go Wrong? Facts About Computer Errors Facts About Computer Errors Error-free software is not possible. Error-free software is not possible. Errors are often caused by more than one factor. Errors are often caused by more than one factor. Errors can be reduced by following good procedures and professional practices. Errors can be reduced by following good procedures and professional practices. Q : How do we distinguish between tolerable or unavoidable errors in software versus careless software development?

3 What Can Go Wrong? The Roles of People in Computer-related Problems: The Roles of People in Computer-related Problems: –Computer User At home or work, users should understand the limitations of computers and the need for proper training and responsible use. At home or work, users should understand the limitations of computers and the need for proper training and responsible use. –Computer Professional Understanding the source and consequences of computer failures is valuable when buying, developing, or managing a complex system. Understanding the source and consequences of computer failures is valuable when buying, developing, or managing a complex system. –Educated Member of Society Personal decisions and political, social, and ethical decisions depend on understanding computer risks. Personal decisions and political, social, and ethical decisions depend on understanding computer risks.

4 What Can Go Wrong? Categories of Computer Errors and Failures Categories of Computer Errors and Failures –Problems for Individuals: usually in their role as consumers. usually in their role as consumers. who are incorrectly identified by inaccurate law enforcement databases. who are incorrectly identified by inaccurate law enforcement databases. –System Failures: affecting large numbers of people and/or costing large amounts of money. affecting large numbers of people and/or costing large amounts of money. –Safety-Critical Applications: where people may be injured or killed. where people may be injured or killed.

5 What Can Go Wrong? Problems for Individuals Problems for Individuals –Billing Errors Lack of tests for inconsistencies and inappropriate amounts. Lack of tests for inconsistencies and inappropriate amounts. –Database Accuracy Problems Incorrect information resulting in wrongful treatment or acts. Incorrect information resulting in wrongful treatment or acts. Q : Describe a computer error or failure that has affected you.

6 What Can Go Wrong? Problems for Individuals (cont ’ d) Problems for Individuals (cont ’ d) –Causes: Large population. Large population. Human common sense not part of automated processing. Human common sense not part of automated processing. Overconfidence in the accuracy of data from a computer. Overconfidence in the accuracy of data from a computer. Errors in data entry. Errors in data entry. Information not updated or corrected. Information not updated or corrected. Lack of accountability for errors. Lack of accountability for errors.

7 What Can Go Wrong? System Failures System Failures –Communications: Telephone, online, and broadcast services. Telephone, online, and broadcast services. –Business: Inventory and management software. Inventory and management software. –Financial: Stock exchange, brokerages, banks, etc.. Stock exchange, brokerages, banks, etc.. –Transportation: Reservations, ticketing, and baggage handling. Reservations, ticketing, and baggage handling.

8 What Can Go Wrong? System Failures (cont ’ d) System Failures (cont ’ d) –Causes: Insufficient testing and debugging time. Insufficient testing and debugging time. Significant changes in specifications (during and after project begun). Significant changes in specifications (during and after project begun). Overconfidence in system. Overconfidence in system. Mismanagement of the project. Mismanagement of the project. Q : Describe a recent system failure that affected many people or resulted in a great monetary loss.

9 What Can Go Wrong? Safety-Critical Applications Safety-Critical Applications –Military –Power Plants –Aircraft –Trains –Automated Factories –Medicine – … others.

10 What Can Go Wrong? Safety-Critical Applications Safety-Critical Applications –Causes: Overconfidence. Overconfidence. Lack of override features. Lack of override features. Insufficient testing. Insufficient testing. Sheer complexity of system. Sheer complexity of system. Mismanagement. Mismanagement. Q : What activities do you participate in that are controlled by safety-critical applications?

11 Case Study: The Therac-25 The Therac-25 was a software-controlled radiation-therapy machine used to treat people with cancer. The Therac-25 was a software-controlled radiation-therapy machine used to treat people with cancer. –Overdoses of radiation Normal dosage is 100 – 200 rads. Normal dosage is 100 – 200 rads. It is estimated that 13,000 and 25,000 rads were given to six people. It is estimated that 13,000 and 25,000 rads were given to six people. Three of the six people died. Three of the six people died. Q : What determines whether the risks associated with a safety-critical application are acceptable?

12 Case Study: The Therac-25 Therac-25 Radiation Overdose Therac-25 Radiation Overdose –Multiple Causes: Poor safety design. Poor safety design. Insufficient testing and debugging. Insufficient testing and debugging. Software errors. Software errors. Lack of safety interlocks. Lack of safety interlocks. Overconfidence. Overconfidence. Inadequate reporting and investigation of accidents. Inadequate reporting and investigation of accidents.

13 Increasing Reliability and Safety What Goes Wrong? What Goes Wrong? –Computer Systems Fail Because: The job they are doing is inherently difficult, and The job they are doing is inherently difficult, and The job is done poorly. The job is done poorly. –Compounding the Reliability Issue: Developers and users exhibit overconfidence in the system. Developers and users exhibit overconfidence in the system. Reused system software may not work in different environments. Reused system software may not work in different environments. Q : Identify the elements needed as an incentive to increase reliability and safety.

14 Increasing Reliability and Safety Professional Techniques Professional Techniques Follow good software practices. Follow good software practices. Exhibit professional responsibility at all levels of development and use. Exhibit professional responsibility at all levels of development and use. Construct well-designed user interfaces. Construct well-designed user interfaces. Take human factors into account. Take human factors into account. Include built-in redundancy. Include built-in redundancy. Incorporate self-checking where appropriate. Incorporate self-checking where appropriate. Follow good testing principals and techniques. Follow good testing principals and techniques. Q : What human interface features should be considered for ordinary business applications?

15 Increasing Reliability and Safety Law and Regulation Law and Regulation –Criminal and Civil Penalties: to recover loss from faulty or unsafe systems. to recover loss from faulty or unsafe systems. –Liability and Civil Laws: to provide incentives to produce reliable and safe systems. to provide incentives to produce reliable and safe systems. –Warranties: to guarantee a certain level of quality. to guarantee a certain level of quality. –Federal or State Regulations: to protect the public. to protect the public.

16 Increasing Reliability and Safety Law and Regulation (cont ’ d) Law and Regulation (cont ’ d) –Database Accuracy Enforcement: to protect the public from inaccurate information maintained by private companies and government. to protect the public from inaccurate information maintained by private companies and government. –Mandatory Licensing of Software Developers: to ensure proper training, competency, and continuing education. to ensure proper training, competency, and continuing education. Q : How can consumers protect themselves from faulty software?

17 Perspectives on Failures, Dependence, Risk, and Progress Failures Failures What are acceptable rates of failures? What are acceptable rates of failures? How accurate should software be? How accurate should software be? Dependence Dependence How dependent on computer systems are our ordinary activities? How dependent on computer systems are our ordinary activities? How useful are computer systems to our ordinary activities? How useful are computer systems to our ordinary activities? Risk and Progress Risk and Progress How do new technologies become safer? How do new technologies become safer? Can progress in software safety keep up with the pace of change in computer technology? Can progress in software safety keep up with the pace of change in computer technology?

18 Computer Models Points to Consider: Points to Consider: Models are simplifications of either physical or intangible systems. Models are simplifications of either physical or intangible systems. Those who design and develop models must be honest and accurate with results. Those who design and develop models must be honest and accurate with results. Computer professionals and the general public must be able to evaluate the claims of the developers. Computer professionals and the general public must be able to evaluate the claims of the developers. Q : What problems in your community have been or could be studied with computer models?

19 Computer Models Evaluating Models Evaluating Models –Why Models Might Not Be Accurate: Developers have incomplete knowledge of the system being modeled. Developers have incomplete knowledge of the system being modeled. Data might be incomplete or inaccurate. Data might be incomplete or inaccurate. Power of the computer might be inadequate. Power of the computer might be inadequate. Variables are difficult to numerically quantify. Variables are difficult to numerically quantify. Political and economic motivation to distort results. Political and economic motivation to distort results. Q : For each item above, give an example of a model or simulation that was inaccurate.

20 Computer Models Evaluating Models (cont ’ d) Evaluating Models (cont ’ d) –Regarding the Car-Crash Models Described in the Text: How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? What are the assumptions and simplifications in the model? What are the assumptions and simplifications in the model? Do the results or predictions correspond with the real world? Do the results or predictions correspond with the real world?

21 Computer Models Evaluating Models (cont ’ d) Evaluating Models (cont ’ d) –Regarding the Climate Models Discussed in the Text: How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? How well do the modelers understand the system and/or materials being studied? How accurate and complete are the data? What are the assumptions and simplifications in the model? What are the assumptions and simplifications in the model? Do the results or predictions correspond with the real world? Do the results or predictions correspond with the real world?