1. Ethics and Software Reliability
Ethics and Software Reliability

2. How we catch bugs: V & V activities
Verification answers: Did we build the system right?
Errors in design (wrt the requirements spec)?
Errors in implementation (wrt the requirements spec)?
Validation answers: Did we build the right system?
Errors in the requirements spec?

3. How might we verify/validate a program?
Prove correct (w/ formal math)?
Nope. Too difficult, even for some small programs
Prove a model correct?
Nope. Too difficult to get small enough model that captures all relevant behavior
Run the program to see if it works?
Yup. Testing, that’s what we do

4. The Testing Problem: How to choose small set of test cases that reveal all errors?
Fundamental research problem
Essentially unsolvable in general case

5. Does all “real” software contain bugs?
Yup. No software is 100% reliable
Let’s consider some examples of reliability failures

6. Financial software failures
Knight Capital loses $440 mil.:
L.A. teachers’ pay:
Phone bill for $57, (Minnesota, 2001)
Beef producers lose $15-$20 million due to bad price reports

7. Data entry/retrieval failures
Bad background checks:
Thousands of voters forbidden to vote because computer miscategorized them as felons (Florida, 2000)
Numerous false arrests due to errors in crime database (NCIC)

8. Software system failures
Mars Surveyor failure (2007):
Therac-25 ( ): multiple radiation overdoses/deaths
Patriot missile system: 28 soldiers killed by SCUD; Patriot never even fires (1991)
Ariane 5 rocket: reuse error causes self-destruct, costing $500 mil.

9. Software unreliability leads to moral and ethical questions
Morality: Rules of conduct
What people ought and ought not do in various situations
Different societies may have different morals
Ethics: Rational examination of people’s moral beliefs and behaviors
Reason from facts to commonly accepted values
Ethical theories: frameworks for reasoning
Examples: Subjective Relativism, Cultural Relativism, Divine Command Theory, Ethical Egoism, Kantaniasm, Utilitarianism…

10. Morality and Ethics
Society is like a town full of people driving cars.
Morality is the road network.
People ought to keep their cars on the roads.
Those who “do ethics” are in balloons.
They can evaluate:
individual roads (moral guidelines)
quality of road network (moral system)
They can judge
if drivers are staying on the roads (acting morally)
or taking shortcuts (acting immorally)
They can propose new road networks (alternative moral systems).
From Quinn (2013), p53

11. Ethical Question
Amazon error (UK, 2003): iPaq price listed as 7 EUR instead of 275 EUR
Amazon refuses to honor sales—were they wrong?
Utilitarian view: Seller must always honor advertised price
Kantian view: Consumers recognized error and did not act in “good faith”

12. Ethical Question
Should Therac-25 developers be held morally responsible for deaths?
Moral responsibility requires:
Causal condition: actions caused harm
Mental condition: actions intended
Unintended harm counts if due to carelessness, recklessness, or negligence
Quinn concludes “yes” because machine lacked devices to catch/prevent overdose and failed to thoroughly verify overdose wasn’t possible

13. If perfect software is impossible, what sort of warranties should companies provide?
Shrinkwrap warranties:
90-day replacement (haha)/money back if softw. fails
Few guarantees (you’ll be able to install it)
Maker has no liability (for harm to your business)
“Don’t blame us if it’s full of bugs”
Are warranties enforceable?
Unfair warranties could violate laws
Court rulings have gone both ways
Should manufacturers be held liable for defects?
Should consumers have to pay for bug fixes?

14. Hypothetical Question
Medick Inc. developing fancy new mobile software for nurses and other hospital workers
Medick told customers it would be ready next week
Software still contains lots of bugs
Competitor will release new product in a few weeks
If competitor gets to market first, Medick will likely go under
What course of action should Medick take?
Perspectives: software engineers, sales force, managers, venture capitalists, and nurses

15. Summary All “software” contains bugs
Summary
All “software” contains bugs
Knowing that perfection is impossible, how to set consumer expectations and assign blame when things go wrong?
Leads to moral and ethical questions

16. Question Anyone been a beta tester?
What did it teach you about computer reliability?