Ethics and Software Reliability http://flic.kr/p/doFGYA Ethics and Software Reliability

How we catch bugs: V & V activities Verification answers: Did we build the system right? Errors in design (wrt the requirements spec)? Errors in implementation (wrt the requirements spec)? Validation answers: Did we build the right system? Errors in the requirements spec?

How might we verify/validate a program? Prove correct (w/ formal math)? Nope. Too difficult, even for some small programs Prove a model correct? Nope. Too difficult to get small enough model that captures all relevant behavior Run the program to see if it works? Yup. Testing, that’s what we do

The Testing Problem: How to choose small set of test cases that reveal all errors? Fundamental research problem Essentially unsolvable in general case

Does all “real” software contain bugs? Yup. No software is 100% reliable Let’s consider some examples of reliability failures

Financial software failures Knight Capital loses $440 mil.: http://www.npr.org/2012/08/18/159082822/in-wall-street-2-0-computers-are-king L.A. teachers’ pay: http://www.npr.org/templates/story/story.php?storyId=14869684 Phone bill for $57,346.20 (Minnesota, 2001) Beef producers lose $15-$20 million due to bad price reports

Data entry/retrieval failures Bad background checks: http://www.nbc.com/news-sports/msnbc-video/2012/11/rossen-reports-background-check-mistakes-cost-jobs/ Thousands of voters forbidden to vote because computer miscategorized them as felons (Florida, 2000) Numerous false arrests due to errors in crime database (NCIC)

Software system failures Mars Surveyor failure (2007): http://www.npr.org/templates/story/story.php?storyId=6817176 Therac-25 (1985-1987): multiple radiation overdoses/deaths Patriot missile system: 28 soldiers killed by SCUD; Patriot never even fires (1991) Ariane 5 rocket: reuse error causes self-destruct, costing $500 mil.

Software unreliability leads to moral and ethical questions Morality: Rules of conduct What people ought and ought not do in various situations Different societies may have different morals Ethics: Rational examination of people’s moral beliefs and behaviors Reason from facts to commonly accepted values Ethical theories: frameworks for reasoning Examples: Subjective Relativism, Cultural Relativism, Divine Command Theory, Ethical Egoism, Kantaniasm, Utilitarianism…

Morality and Ethics Society is like a town full of people driving cars. Morality is the road network. People ought to keep their cars on the roads. Those who “do ethics” are in balloons. They can evaluate: individual roads (moral guidelines) quality of road network (moral system) They can judge if drivers are staying on the roads (acting morally) or taking shortcuts (acting immorally) They can propose new road networks (alternative moral systems). From Quinn (2013), p53

Ethical Question Amazon error (UK, 2003): iPaq price listed as 7 EUR instead of 275 EUR Amazon refuses to honor sales—were they wrong? Utilitarian view: Seller must always honor advertised price Kantian view: Consumers recognized error and did not act in “good faith”

Ethical Question Should Therac-25 developers be held morally responsible for deaths? Moral responsibility requires: Causal condition: actions caused harm Mental condition: actions intended Unintended harm counts if due to carelessness, recklessness, or negligence Quinn concludes “yes” because machine lacked devices to catch/prevent overdose and failed to thoroughly verify overdose wasn’t possible

If perfect software is impossible, what sort of warranties should companies provide? Shrinkwrap warranties: 90-day replacement (haha)/money back if softw. fails Few guarantees (you’ll be able to install it) Maker has no liability (for harm to your business) “Don’t blame us if it’s full of bugs” Are warranties enforceable? Unfair warranties could violate laws Court rulings have gone both ways Should manufacturers be held liable for defects? Should consumers have to pay for bug fixes?

Hypothetical Question Medick Inc. developing fancy new mobile software for nurses and other hospital workers Medick told customers it would be ready next week Software still contains lots of bugs Competitor will release new product in a few weeks If competitor gets to market first, Medick will likely go under What course of action should Medick take? Perspectives: software engineers, sales force, managers, venture capitalists, and nurses

Summary All “software” contains bugs http://flic.kr/p/YSY3X Summary All “software” contains bugs Knowing that perfection is impossible, how to set consumer expectations and assign blame when things go wrong? Leads to moral and ethical questions

Question Anyone been a beta tester? What did it teach you about computer reliability?