1. Ethics of Software Testing Thomas LaToza CS 210 Final Presentation 12 / 2 / 2002

2. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Overview Background - facts of testing and bugs Education, training, best practices, and a commitment to reliability All help to improve reliability Don’t solve the problem and still leave Research How much to spend on testing When to release software Most organizations should spend more on testing

3. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Facts of Testing Testing is an activity to evaluate quality and find bugs Testing only proves presence of bugs Testing NEVER proves absence of bugs (unless you could run every line of code for all possible inputs for all possible states in all possible environments with an oracle that knows the correct output and state) Testing is underappreciated None of the top 5 CS departments offer an undergraduate course on testing or building software systems for reliability (other than formal verification) – only taught as part of a software engineering course Often not perceived as challenging and fun Admission of failure – people make mistakes

4. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Finding Bugs Not all bugs are equal. Bugs may be Easier to find than other bugs More important than other bugs (bugs that crash or destroy data vs. bugs that make the interface look bad) Invisible until other bugs are fixed

5. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Fixing Bugs Fixing a bug always has a chance of introducing new bug(s) which may be worse Overall effect of testing depends on ratio of bugs fixed to bugs introduced

6. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Bugs aren’t cheap For users Bugs and glitches cost the economy $59.5 Billion a year (NIST estimate) For software projects Testing cost can easily be 25 – 50% cost of project and can be much more for certified systems

7. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Testing bug curve Bugs Time / Effort / Testing resources

8. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Overview Background - facts of testing and bugs Education, training, best practices, and a commitment to reliability All help to improve reliability Don’t solve the problem and still leave Research How much to spend on testing When to release software Most organizations should spend more on testing

9. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Engineering practices can help prevent bugs Code reviews – highly effective Team members proof read the code and discuss Writing for readability There are two ways of constructing a software design; one way is to make it so simple that there are obviously no deficiencies, and the other way is to make it so complicated that there are no obvious deficiencies. The first method is far more difficult. - C. A. R. Hoare Design by test Write the interface and test case first, then write the functionality Statistical testing Commercial off the shelf software / Open Source / Components Separate Quality Assurance team Betas But none of these replace testing…

10. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Research Problem has (recently) sparked great research interest Recovery Oriented Computing Formal Methods Program Comprehension Software Engineering

11. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Testing Review It’s expensive and time consuming Doesn’t guarantee lack of bugs Developers often don’t like doing it But it’s really important…

12. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Overview Background - facts of testing and bugs Education, training, best practices, and a commitment to reliability All help to improve reliability Don’t solve the problem and still leave Research How much to spend on testing When to release software Most organizations should spend more on testing

13. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Review – Applying Ethical Theories 1. Evaluate information 2. Consider how decision may affect stakeholders 3. Consider what ethics [ethical values] are relevant to situation 4. Determine the best course of action that takes into account relevant values and stakeholders’ interests. (from Lockheed Martin Decision Making Model)

14. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign How much to spend on testing (1. Evaluate Information) Most organizations should spend more on testing But have to stop somewhere – can’t spend all of your budget on testing Question of where on the curve is best There are advantages for being further left and advantages for being further right

15. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign More testing / Less bugs (2. Consider how decision may affect stakeholders) Advantages Developers / Customers – less time working around bugs Customers – less frustration dealing with software Managers – know more about bugs in product and can deliver a higher quality product Company – quality makes software more valuable to customer Company – sooner bugs can be fixed the cheaper they are to fix Disadvantages Customers – more expensive products Company – less features, less compelling reasons for customer to spend money upgrading

16. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign How much to spend on testing (3. Consider what ethics [ethical values] are relevant to situation) (4. Determine best action based on relevant values and stakeholders) Utilitarian – maximize cost / benefit for company and / or customers Value – have a responsibility to provide a quality product by setting a quality standard and sticking to it Libertarian – just enough that it helps company the most (customer fends for himself) Altruism – so that it is best for the customer

17. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign When to release (1. Evaluate Information) Would like to have a higher quality product But have to stop somewhere – may not ever find all of the bugs Again question of where on the curve is best There are advantages for being further left and advantages for being further right

18. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Release later / Less bugs (2. Consider how decision may affect stakeholders) Advantages All of the advantages of more testing / less bugs Managers – know more about quality of the product being delivered Company / customers – higher quality product Disadvantages All of the disadvantages of less testing / more bugs Company – may lose marketshare to competitors Company – no income from product Customer – can’t use product at all Company – may lose credibility for missing deadline

19. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign When to release (3. Consider what ethics [ethical values] are relevant to situation) (4. Determine best action based on relevant values and stakeholders) Utilitarian – maximize cost / benefit for company and / or customers Value – set quality standard vs. keeping a ship date vs. money for company Libertarian – whenever the most customers will buy it Altruism – so that customer will have optimum balance of most time to use it and the best product

20. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Conclusions Many other ethical questions Should a bug be fixed? How much information about testing to release? Do developers and testers have professional standards to uphold? Expectations for quality – too high or too low?

21. © Thomas LaToza - 12/2/2002 – Ethical and Professional Issues in Computing Final Presentation – University of Illinois at Urbana-Champaign Further Reading Economics of Software Verification http://cm.bell-labs.com/cm/cs/who/gerard/gz/paste01.pdf