1. Lecture 4b: Risks and Liabilities of Computer-based Systems
CSCI102 - Introduction to Information Technology B
ITCS905 - Fundamentals of Information Technology

2. Overview Historical examples of software risks
Implications of software complexity
Risk assessment and management

3. Historical Examples Software errors Can KILL Cost MONEY Indirectly
Loss of equipment
Loss of business

4. Software Aids and Abets Murder: 1992
A New Jersey inmate escaped from computer-monitored house arrest in the spring of 1992
He simply removed the rivets holding his electronic anklet together and went off to commit a murder

5. Software Aids and Abets Murder: 1992
A computer detected the tampering
when it called a second computer to report the incident, the first computer received a busy signal and never called back

6. Radiation Machine Kills Four: 1985 to 1987
Faulty software in a Therac-25 radiation-treatment machine resulted in several cancer patients receiving lethal overdoses of radiation

7. Radiation Machine Kills Four: 1985 to 1987
Four patients died

8. Radiation Machine Kills Four: 1985 to 1987
When their families sued, all the cases were settled out of court
There were several errors,
among them the failure of the programmer to detect a race condition (i.e., miscoordination between concurrent tasks)
Faulty software in a Therac-25 radiation-treatment machine made by Atomic Energy of Canada Limited (AECL) resulted in several cancer patients receiving lethal overdoses of radiation
Four patients died.
When their families sued, all the cases were settled out of court. A later investigation by independent scientists Nancy Leveson and Clark Turner found that accidents occurred even after AECL thought it had fixed particular bugs. "A lesson to be learned from the Therac-25 story is that focusing on particular software bugs is not the way to make a safe system," they wrote in their report. "The basic mistakes here involved poor software-engineering practices and building a machine that relies on the software for safe operation.“
In 1986, two cancer patients at the East Texas Cancer Center in Tyler received fatal radiation overdoses from the Therac-25, a computer-controlled radiation-therapy machine. There were several errors, among them the failure of the programmer to detect a race condition (i.e., miscoordination between concurrent tasks).

9. Radiation Machine Kills Four: 1985 to 1987
It was found that found that accidents occurred even after AECL thought it had fixed particular bugs
"A lesson to be learned from the Therac-25 story is that focusing on particular software bugs is not the way to make a safe system”
"The basic mistakes here involved poor software-engineering practices and building a machine that relies on the software for safe operation”
Faulty software in a Therac-25 radiation-treatment machine made by Atomic Energy of Canada Limited (AECL) resulted in several cancer patients receiving lethal overdoses of radiation
Four patients died.
When their families sued, all the cases were settled out of court. A later investigation by independent scientists Nancy Leveson and Clark Turner found that accidents occurred even after AECL thought it had fixed particular bugs. "A lesson to be learned from the Therac-25 story is that focusing on particular software bugs is not the way to make a safe system," they wrote in their report. "The basic mistakes here involved poor software-engineering practices and building a machine that relies on the software for safe operation.“
Item: In 1986, two cancer patients at the East Texas Cancer Center in Tyler received fatal radiation overdoses from the Therac-25, a computer-controlled radiation-therapy machine. There were several errors, among them the failure of the programmer to detect a race condition (i.e., miscoordination between concurrent tasks).

10. Hyphen Costs $80 Million: 1962
A probe launched from Cape Canaveral was set to go to Venus
After takeoff, the unmanned rocket carrying the probe went off course
NASA had to blow up the rocket to avoid endangering lives on earth

11. Hyphen Costs $80 Million: 1962
NASA later attributed the error to a faulty line of Fortran code
“Somehow a hyphen had been dropped from the guidance program loaded aboard the computer, allowing the flawed signals to command the rocket to veer left and nose down
...Suffice it to say, the first U.S. attempt at interplanetary flight failed for want of a hyphen”

12. Hyphen Costs $80 Million: 1962
The vehicle cost more than $80 million, prompting Arthur C. Clarke to refer to the mission as “the most expensive hyphen in history”

13. AT&T Long Distance Service Fails: 1991
In the summer of 1991, telephone outages occurred in local telephone systems in California and along the Eastern seaboard
These breakdowns were all the fault of an error in signalling software

14. AT&T Long Distance Service Fails: 1991
Right before the outages
DSC Communications introduced a bug when it changed three lines of code in the several-million-line signalling program
After this tiny change, nobody thought it necessary to retest the program

15. AT&T Long Distance Service Fails: 1991
These switching errors in AT&T's call-handling computers caused the company's long-distance network to go down for nine hours
The meltdown affected thousands of services and was eventually traced to a single faulty line of code

16. There’s a Hole in the Bucket
Small systems
…form part of larger systems
A fault within a small part could result in a catastrophe later on

17. There’s a Hole in the Bucket
Designers have an ethical responsibility to design the best system possible

18. Bugs Bugs exist because
…humans aren't perfect
Since humans design and program hardware and software, mistakes are inevitable
That's what computer and software vendors tell us, and it's partly true
What they don't say is that software is buggier than it has to be

19. Bugs
Why?
Because time is money, especially in the software industry

20. Bugs
This is how bugs are born

21. Bugs
A software or hardware company sees a business opportunity and starts building a product to take advantage of that
Long before development is finished, the company announces that the product is on the way
A software or hardware company sees a business opportunity and starts building a product to take advantage of that
Long before development is finished, the company announces that the product is on the way
Because the public is (the company hopes) now anxiously awaiting this product, the marketing department fights to get the goods out the door before that deadline

22. Bugs
All the while pressuring the software engineers to add more and more features

23. Bugs
Shareholders and venture capitalists clamour for quick delivery because that's when the company will see the biggest surge in sales
Meanwhile, the quality-assurance division has to battle for sufficient bug-testing time

24. Bugs
“The simple fact is that you get the most revenues at the release of software,”
“The faster you bring it out, the more money you make. You can always fix it later, when people howl. It's a fine line when to release something, and the industry accepts defects“
 "The simple fact is that you get the most revenues at the release of software," says Bruce Brown, the founder of BugNet, a newsletter that has chronicled software bugs and fixes since "The faster you bring it out, the more money you make. You can always fix it later, when people howl. It's a fine line when to release something, and the industry accepts defects"

25. What Is Risk Assessment and Management?
Risk and uncertainty are fundamental elements of modern life
They are ever present in the actions of human beings and they are frequently magnified in large-scale technological systems
Risk and uncertainty must be managed effectively to protect people from injury and to permit the development of reliable, high-quality products

26. What Is Risk Assessment and Management?
Risk is often defined as a measure of the probability and severity of adverse effects

27. What Is Risk Assessment and Management?
In risk assessment, the analyst often attempts to answer the following set of triplet questions
What can go wrong?
What is the likelihood that it would go wrong?
What are the consequences?

28. What Is Risk Assessment and Management?
Answers to these questions help risk analysts identify, measure, quantify, and evaluate risks and their consequences and impacts

29. What Is Risk Assessment and Management?
Risk management builds on the risk assessment process by seeking answers to a second set of three questions
What can be done?
What options are available and what are their associated trade-offs in terms of all costs, benefits, and risks?
What are the impacts of current management decisions on future options?

30. What Is Risk Assessment and Management?
To be effective and meaningful, risk management must be an integral part of the overall management of a system
This is particularly important in the management of technological systems, where the failure of the system can be caused by the failure of the hardware, the software, the organization, or the humans