1. Trust, Safety and Reliability
CS 340, Spring 2013
Ch. 5 in Ethics in a Computing Culture

2. Computer errors A range of possible consequences Examples:
Seconds lost, other inconvenience
Data lost
$ - $$$
Injuries and deaths
Examples:
Disfranchised voters
False arrests
Price & billing errors
Air traffic control, airplane control

3. The Apocalypses That Might Have Been

4. Apocolypse cont’d

5. Ch. 5 Topics Computer reliability can be a matter of life or death
Exploring the various ways people trust computer and technology to perform tasks for them.
Examining failures
Approaches to minimize risks

6. Causes of Computer Failure
A computer might fail to meet expectations by:
Hardware errors (malfunction)
Software errors (bugs)
Being programmed to solve the wrong problem (programmers fail to deliver client expectations)
Misuse (a computer is provided erroneous data)
Communication failure (human misunderstands a computer prompt)
Malice (hackers)
Ethics in a Computing Culture

7. Hardware errors
Physical components of the system

8. Software Errors
“Bugs”
Bug v. mistake debate

9. Bugs and Public Safety
Safety-critical software: software that may affect someone’s safety if it fails to work properly
Decision point: a place in computer code where the next instruction executed depends on input data
Control programs: programs that control some sort of machinery
Ethics in a Computing Culture

10. Bugs and Public Safety (continued)
Real time: a program must do something within a specific amount of time
Multiprocess: programs that execute at the same time as one or more other programs
Ethics in a Computing Culture

11. Bugs and Public Safety (continued)
Many executives of software production companies, as well as several famous computer scientists, have asserted that the most important skill needed by a software developer is the ability to communicate.
Would more extensive testing of the software have prevented the Ariane flight failure?
Ethics in a Computing Culture

12. Bugs and Public Safety (continued)
Brooks’s Law: “Adding people to a late project makes it later.”
Adding new people adds new lines of communication, which increase the chances of miscommunication or missed communication
Ethics in a Computing Culture

13. Software errors Pages 152- 154 Pages 155-156 In 2005, Ariane 5
ariane 5 explosion – YouTube
Floating point number storage
Arithmetic overflow
Pages
In 1994, Intel P5 Pentium chip
Certain floating point numbers when divided produced invalid results

14. Questions for the Pentium Floating Point Divide
Did Intel have a moral obligation to replace the defective Pentium 5 chips, even for those users who had no need for high precision?
When employees of Intel first discovered the bug, did they have an ethical responsibility to make the problem public?
Ethics in a Computing Culture

15. Computer Solves the Wrong Problem
Complexity of programming: garbage in, garbage out
Gemini 5, 105 miles off landing target:
Earth’s rotation is not 360° in 24 hr but °
Friendship 7, 40 miles off landing target:
Failure to take into account weight loss in aircraft due to use of consumables

16. Misuse The Hernandez situation:
Emiliano Hernandez was pulled over running a stop sign
When the officer ran his license the officer was convinced that he was dealing with a wanted man, “Enrique” Hernandez
Same birthday, similar height, weight, tattoo.
Officer was convinced that the name difference was a mistake in the system. It was not.

17. Communication Failure
Misunderstandings
Of what the system is capable of or how it works
Book example -refueling
Electronic voting machines:
After the "hanging chad" controversy of the 2000 election, Congress passed a federal law that gave states funding to replace their punch card and lever voting systems with electronic voting machines

18. Electronic voting problems/concerns
North Carolina 2004
Voting machine manufacturer said the electronic voting machine could handle 10,500 votes each
Reality – could only hold 3,005 votes
Result – lost 4,530 votes
Many voting machines do not create a paper trail
Voting machines used by as many as a quarter of American voters in 2012 can be hacked with just $26 in parts and an 8th grade science education
Vulnerability Assessment Team at Argonne National Laboratory 
The experts say the hack could change voting results while leaving absolutely no trace of the manipulation behind

19. Malice With ill intent people destroy or modify computer systems.
Can be for-profit crime, terrorism, or warfare (this topic will be covered in Tuesday’s class)

20. Intrinsic & Extrinsic software
Software that is part of a completed product
Extrinsic:
Loaded onto the computer or machine of the user, user directly encounters

21. Important Case Study: The Therac-25
Radiation therapy machine
Typical system malfunctions number 40x day

22. The Therac-25 case cont’d
20 month period, overdoses to 6 patients, directly killing 3.
Previous models, 6 & 20
Differences to the 25?
Chronology of accidents
Please see

23. What was wrong with the Therac-25 programming?
2 modes
X-ray: high intensity beam deflected by tungsten target
Electron: removes tungsten & reduces beam intensity by factor of 100
Quickly changing* (data entry editing) between mode resulted in electron mode not dropping the beam intensity
* If the operator was able to edit and start < 8 seconds
Use of a Race condition – 2 or more tasks sharing a variable, order that each is encountered can affect behavior of the program
The Therac-25 no longer had the hardware safety feature

24. Therac 25 Problems No fail safe No dose reporting
Complicated programming
Re-use of code

25. Who had moral and/or legal responsibility?
Harm clearly shown. Was there intent? No
So we consider tort of negligence
Defining negligence from law.com
n. failure to exercise the care toward others which a reasonable or prudent person would do in the circumstances, or taking action which such a reasonable person would not
Must prove:
“a) that the party alleged to be negligent had a duty to the injured party-specifically to the one injured or to the general public, b) that the defendant's action (or failure to act) was negligent-not what a reasonably prudent person would have done, c) that the damages were caused ("proximately caused") by the negligence. An added factor in the formula for determining negligence is whether the damages were "reasonably foreseeable" at the time of the alleged carelessness”

26. Extrinsic Software Failures
With these examples, companies or individuals are buying software for what it can do for them.
Disclaiming liability for problems through warranties

27. Software Warranties Limiting liability to: Accepting no liability for
A refund of the purchase price
Repair of the software product
Accepting no liability for
Business losses arising out of the use of the product
Enforceability of these disclaimers?
UCC & the Magnuson-Moss Warranty Act
Mix of case law

28. Extrinsic Business Software Failures
NCR’s Warehouse Manager
Warehouse Manager was an inventory program. It was developed for a different operating system than it was deployed for.
The “deadly embrace”
NCR continued selling it, claiming 200 successful installs, but that was actually on other op sys.
When problems reported, told customers that was “unique”

29. NCR cont’d Hopper bought the system.
Did not get honest disclosure from NCR.
NCR sold it to him after product had been discontinued.
Errors about inventory and pricing resulted.
$114 item listed for 54 cents; $17 item listed as on sale for $30.
Hopper was operating a successful co., but after adoption of WM income was half.
Processes took to long & were inaccurate. Inventory inaccuracies

30. NCR conclusion Hopper tried to sue NCR for $4.2 million.
However, the sales agreement signed by Hopper stated that in the event of problems, NCR was only responsible for the original cost of the software minus the depreciation of the equipment.
The agreement also had an arbitration clause, so the court refused the case.

31. ProCD v. Zeidenberg ProCD selling mailing list generating software
2 prices, 1 for personal 1 for business
Zeidenberg bought it as personal and created a mailing list sales business using the product.
ProCD sued saying this violated terms of the license
Terms not found on box but on click thru agmt
Ct found that Zeidenberg could be held to those terms of the click through agreement and was in violation of the software license.

32. Mortenson v. Timberline Software
Precison Bid software
Used it and created bid $1.95 million too low
Licensing disclaimed business losses related to use of the software in excess of licensing fee
Timberline was aware of bug, did not send fix to Mortenson.
Ct. found the Timberline was not liable b/c licensing agmt. properly limited liability.