Lecture 4b: Risks and Liabilities of Computer-based Systems CSCI102 - Introduction to Information Technology B ITCS905 - Fundamentals of Information Technology

Overview Historical examples of software risks Implications of software complexity Risk assessment and management

Historical Examples Software errors Can KILL Cost MONEY Indirectly Loss of equipment Loss of business

Software Aids and Abets Murder: 1992 A New Jersey inmate escaped from computer-monitored house arrest in the spring of 1992 He simply removed the rivets holding his electronic anklet together and went off to commit a murder

Software Aids and Abets Murder: 1992 A computer detected the tampering when it called a second computer to report the incident, the first computer received a busy signal and never called back

Radiation Machine Kills Four: 1985 to 1987 Faulty software in a Therac-25 radiation-treatment machine resulted in several cancer patients receiving lethal overdoses of radiation

Radiation Machine Kills Four: 1985 to 1987 Four patients died

Radiation Machine Kills Four: 1985 to 1987 When their families sued, all the cases were settled out of court There were several errors, among them the failure of the programmer to detect a race condition (i.e., miscoordination between concurrent tasks) Faulty software in a Therac-25 radiation-treatment machine made by Atomic Energy of Canada Limited (AECL) resulted in several cancer patients receiving lethal overdoses of radiation Four patients died. When their families sued, all the cases were settled out of court. A later investigation by independent scientists Nancy Leveson and Clark Turner found that accidents occurred even after AECL thought it had fixed particular bugs. "A lesson to be learned from the Therac-25 story is that focusing on particular software bugs is not the way to make a safe system," they wrote in their report. "The basic mistakes here involved poor software-engineering practices and building a machine that relies on the software for safe operation.“ In 1986, two cancer patients at the East Texas Cancer Center in Tyler received fatal radiation overdoses from the Therac-25, a computer-controlled radiation-therapy machine. There were several errors, among them the failure of the programmer to detect a race condition (i.e., miscoordination between concurrent tasks).

Radiation Machine Kills Four: 1985 to 1987 It was found that found that accidents occurred even after AECL thought it had fixed particular bugs "A lesson to be learned from the Therac-25 story is that focusing on particular software bugs is not the way to make a safe system” "The basic mistakes here involved poor software-engineering practices and building a machine that relies on the software for safe operation” Faulty software in a Therac-25 radiation-treatment machine made by Atomic Energy of Canada Limited (AECL) resulted in several cancer patients receiving lethal overdoses of radiation Four patients died. When their families sued, all the cases were settled out of court. A later investigation by independent scientists Nancy Leveson and Clark Turner found that accidents occurred even after AECL thought it had fixed particular bugs. "A lesson to be learned from the Therac-25 story is that focusing on particular software bugs is not the way to make a safe system," they wrote in their report. "The basic mistakes here involved poor software-engineering practices and building a machine that relies on the software for safe operation.“ Item: In 1986, two cancer patients at the East Texas Cancer Center in Tyler received fatal radiation overdoses from the Therac-25, a computer-controlled radiation-therapy machine. There were several errors, among them the failure of the programmer to detect a race condition (i.e., miscoordination between concurrent tasks).

Hyphen Costs $80 Million: 1962 A probe launched from Cape Canaveral was set to go to Venus After takeoff, the unmanned rocket carrying the probe went off course NASA had to blow up the rocket to avoid endangering lives on earth

Hyphen Costs $80 Million: 1962 NASA later attributed the error to a faulty line of Fortran code “Somehow a hyphen had been dropped from the guidance program loaded aboard the computer, allowing the flawed signals to command the rocket to veer left and nose down ...Suffice it to say, the first U.S. attempt at interplanetary flight failed for want of a hyphen”

Hyphen Costs $80 Million: 1962 The vehicle cost more than $80 million, prompting Arthur C. Clarke to refer to the mission as “the most expensive hyphen in history”

AT&T Long Distance Service Fails: 1991 In the summer of 1991, telephone outages occurred in local telephone systems in California and along the Eastern seaboard These breakdowns were all the fault of an error in signalling software

AT&T Long Distance Service Fails: 1991 Right before the outages DSC Communications introduced a bug when it changed three lines of code in the several-million-line signalling program After this tiny change, nobody thought it necessary to retest the program

AT&T Long Distance Service Fails: 1991 These switching errors in AT&T's call-handling computers caused the company's long-distance network to go down for nine hours The meltdown affected thousands of services and was eventually traced to a single faulty line of code

There’s a Hole in the Bucket Small systems …form part of larger systems A fault within a small part could result in a catastrophe later on

There’s a Hole in the Bucket Designers have an ethical responsibility to design the best system possible

Bugs Bugs exist because …humans aren't perfect Since humans design and program hardware and software, mistakes are inevitable That's what computer and software vendors tell us, and it's partly true What they don't say is that software is buggier than it has to be

Bugs Why? Because time is money, especially in the software industry

Bugs This is how bugs are born

Bugs A software or hardware company sees a business opportunity and starts building a product to take advantage of that Long before development is finished, the company announces that the product is on the way A software or hardware company sees a business opportunity and starts building a product to take advantage of that Long before development is finished, the company announces that the product is on the way Because the public is (the company hopes) now anxiously awaiting this product, the marketing department fights to get the goods out the door before that deadline

Bugs All the while pressuring the software engineers to add more and more features

Bugs Shareholders and venture capitalists clamour for quick delivery because that's when the company will see the biggest surge in sales Meanwhile, the quality-assurance division has to battle for sufficient bug-testing time

Bugs “The simple fact is that you get the most revenues at the release of software,” “The faster you bring it out, the more money you make. You can always fix it later, when people howl. It's a fine line when to release something, and the industry accepts defects“  "The simple fact is that you get the most revenues at the release of software," says Bruce Brown, the founder of BugNet, a newsletter that has chronicled software bugs and fixes since 1994. "The faster you bring it out, the more money you make. You can always fix it later, when people howl. It's a fine line when to release something, and the industry accepts defects"

What Is Risk Assessment and Management? Risk and uncertainty are fundamental elements of modern life They are ever present in the actions of human beings and they are frequently magnified in large-scale technological systems Risk and uncertainty must be managed effectively to protect people from injury and to permit the development of reliable, high-quality products

What Is Risk Assessment and Management? Risk is often defined as a measure of the probability and severity of adverse effects

What Is Risk Assessment and Management? In risk assessment, the analyst often attempts to answer the following set of triplet questions What can go wrong? What is the likelihood that it would go wrong? What are the consequences?

What Is Risk Assessment and Management? Answers to these questions help risk analysts identify, measure, quantify, and evaluate risks and their consequences and impacts

What Is Risk Assessment and Management? Risk management builds on the risk assessment process by seeking answers to a second set of three questions What can be done? What options are available and what are their associated trade-offs in terms of all costs, benefits, and risks? What are the impacts of current management decisions on future options?

What Is Risk Assessment and Management? To be effective and meaningful, risk management must be an integral part of the overall management of a system This is particularly important in the management of technological systems, where the failure of the system can be caused by the failure of the hardware, the software, the organization, or the humans