1. University of Southern California Center for Systems and Software Engineering Software Engineering Ethics Barry Boehm CS 577a 2009 © USC-CSSE1

2. University of Southern California Center for Systems and Software Engineering Outline Definitions and context –Power to do public harm or good –ACM/IEEE Software Engineering Code of Ethics Principles and examples –Rawls’ Theory of Justice –Relation to stakeholder win-win –Case study: Mercy Hospital Integrating ethics into daily software engineering practices –ICM/VBSE/MBASE –CS 577 ethics situations © USC-CSSE2

3. University of Southern California Center for Systems and Software Engineering Definition of “Ethics” -Webster, 1993 The discipline dealing with what is good and bad –And with moral duty and obligation A theory, system, or set of moral principles or values The principles of conduct governing an individual or group –Professional ethics © USC-CSSE3

4. University of Southern California Center for Systems and Software Engineering Context Software engineers have increasing power to do public harm or good –Intellectual property, privacy, confidentiality, quality of work, fairness, liability, risk disclosure, conflict of interest, unauthorized access Professional societies have developed codes of ethics Hard to integrate value-based ethics into value- neutral software engineering practices ICM/VBSE/MBASE enable ethics integration –And buildup of trust © USC-CSSE4

5. University of Southern California Center for Systems and Software Engineering © USC-CSSE5 Shared Commitments are Needed to Build Trust New partnerships are increasingly frequent –They start with relatively little built-up trust Group performance is built on a bedrock of trust –Without trust, partners must specify and verify details Trust is built on a bedrock of honored commitments Once trust is built up, processes can become more fluid –But need to be monitored as situations change

6. University of Southern California Center for Systems and Software Engineering © USC-CSSE6 Elements of Commitments adapted from [Humphrey, 1989] Made willingly, subject to assumptions Not made lightly Agreement on what is to be done, by whom, and when Openly and publicly stated Committers make every effort to meet commitments Unforeseen conflicts and invalidity of assumptions are addressed early, and new commitments negotiated

7. University of Southern California Center for Systems and Software Engineering Power to Do Public Harm or Good – I Intellectual Property: use without credit; use copyrighted material Privacy: credit, health, personal information Confidentiality: competitive information, political sensitivity Quality of work: many dimensions; see table © USC-CSSE7

8. University of Southern California Center for Systems and Software Engineering Example: Confidentiality Government agency hires company to support SW procurement –Provides data under nondisclosure agreement Employee and company consultant prepare cost estimate –Employee: “ I don’t see how anyone can do all this for $8M” Consultant provides $8M target cost to some bidders Government agency angry with company for leak –Whose fault? How could it be avoided? © USC-CSSE8

9. University of Southern California Center for Systems and Software Engineering Quality Concerns Vary by Stakeholders Role © USC-CSSE9 Mission - Protection criticaluncrit. Safety ** Security*** Privacy** * Robustness Reliability ** ** Availability ** ** Survivability ** Quality of Service Performance ** * ** Accuracy, Consistency** * * Accessibility, ease of use;* ** * difficulty of misuse Evolvability ** ** Interoperability ** * Correctness ** Cost * ** Schedule * ** Reusability **** Acquirers Administrators Developers, Maintainers System Controllers Information Consumers Info Brokers System Dependents Info Suppliers Stakeholder Classes **Critical *Significant 0 Insignificant or indirect

10. University of Southern California Center for Systems and Software Engineering Power to Do Public Harm or Good - II Fairness: equality of opportunity/treatment; fair reward system Liability: accountability; parity of authority and responsibility Risk Disclosure: safety tests, COTS capabilities; schedule slips Conflict of Interest: source selection; personnel or product reviews Unauthorized Access: reading, copying, modifying; denial of service © USC-CSSE10

11. University of Southern California Center for Systems and Software Engineering Examples: Fairness Enron software to schedule power outages, raise prices –Suppose you had been asked to develop it? Urban fire dispatching system –Inefficient old system caused $700M property loss –New-system spec. includes dispatching algorithm to minimize property loss Any fairness issues? © USC-CSSE11

12. University of Southern California Center for Systems and Software Engineering Fairness and Accountability Issues: Fire Dispatching System Dispatch to minimize value of property loss –Neglect safety, least-advantaged property owners English-only dispatcher service –Neglect least-advantaged immigrants Minimal recordkeeping –Reduced accountability Tight budget; design for nominal case –Neglect reliability, safety, crisis performance © USC-CSSE12

13. University of Southern California Center for Systems and Software Engineering CS 577 Ethics Accountability Honoring commitments to CS 577b –Team DCR Life Cycle Plan for 577b should identify 577b continuing team members and roles. –If you signed that you will continue in 577b in the basic 577a questionnaire, we are expecting you to honor your commitment. –If you are considering not honoring your commitment, please meet with me as soon as possible. © USC-CSSE13

14. University of Southern California Center for Systems and Software Engineering Example: Safety Tests Your company is delivering a drug prescription fulfillment system –Reusing software from a warehouse inventory system You are the quality assurance manager –With company responsibility for certifying product safety The software has passed all the contracted tests –But many off-nominal conditions untested –Some have shown unsafe outcomes –You feel more off-nominal testing if necessary Company president says if you don’t certify safety by delivery date, company may go out of business –What should you do? © USC-CSSE14

15. University of Southern California Center for Systems and Software Engineering ACM/IEE Software Engineering Code of Ethics -Table of Contents 1.Products: achievable goals, realistic estimates, high quality 2.Public: safety, respect of diversity, public interest first 3.Judgment: objectivity, no bribes or conflicts of interest 4.Client and Employer: no employer-adverse interests, surface problems 5.Management: fair, ethical work rules, due process for violations 6.Profession: support profession and ethics code, don’t misrepresent software 7.Colleagues: credit colleagues’ work, give colleagues a fair hearing 8.Self: improve your technical and ethical knowledge and practices © USC-CSSE15

16. University of Southern California Center for Systems and Software Engineering Code of Ethics 2. Public 2.01 Disclose any software-related dangers 2.02 Approve only safe, well tested software 2.03 Only sign documents in area of competence 2.04 Cooperate on matters of public concern 2.05 Produce software that respects diversity 2.06 Be fair and truthful in all matters 2.07 Always put the public’s interest first 2.08 Donate professional skills to good causes 2.10 Accept responsibility for your own work © USC-CSSE16

17. University of Southern California Center for Systems and Software Engineering Code of Ethics 4. Client and Employer 4.01 Provide services only where competent 4.02 Ensure resources are authentically approved 4.03 Only use property as authorized by the owner 4.04 Do not use illegally obtained software 4.05 Honor confidentiality of information 4.06 Raise matters of social concern 4.07 Inform when a project becomes problematic 4.08 Accept no detrimental outside work 4.09 Represent no interests adverse to your employer © USC-CSSE17

18. University of Southern California Center for Systems and Software Engineering Outline Definitions and context –Power to do public harm or good –ACM/IEEE Software Engineering Code of Ethics Principles and examples –Rawls’ Theory of Justice –Relation to stakeholder win-win –Case study: Mercy Hospital Integrating ethics into daily software engineering practices –ICM/VBSE/MBASE –CS 577 ethics situations © USC-CSSE18

19. University of Southern California Center for Systems and Software Engineering Rawls’ Theory of Justice (1971) - Following Collins et al., “How Good Is Good Enough?” Comm.ACM, Jan. 1994 Fair rules of conduct Principles of justice Participants and obligations –Provider (developer) –Buyer (acquirer) –User(s) –Penumbra (general public) Negotiate mutually satisfactory (win-win) agreements © USC-CSSE19

20. University of Southern California Center for Systems and Software Engineering Rawls’ Theory of Justice - II Fair rules of conduct –Negotiation among interested parties –Veil of ignorance (about what affects whom) –Rationality Principles –Least Advantaged - don’t increase harm to them Harm = probability x magnitude (~risk exposure) –Risking harm - don’t risk increasing harm Don’t use “low-threat” software in “high-threat” context –Publicity test - defensible with honor before an informed public Use for difficult cost-benefit tradeoffs © USC-CSSE20

21. University of Southern California Center for Systems and Software Engineering Obligations of the Software Provider © USC-CSSE21

22. University of Southern California Center for Systems and Software Engineering Obligations of the Software Buyer © USC-CSSE22

23. University of Southern California Center for Systems and Software Engineering Obligations of the Software User © USC-CSSE23

24. University of Southern California Center for Systems and Software Engineering Obligations of the Software Penumbra © USC-CSSE24

25. University of Southern California Center for Systems and Software Engineering Case Study: Mercy Hospital Pharmacy System -Collins et al., 1994 Growing hospital –Manual pharmacy information system reaching overload Spec developed for PC-based information system –Rachel: VP, Records & Automation –George: Chief Pharmacist System developed by consultants –Hired by George –Rachel: test procedures –Based on mature warehouse inventory system –Budgeted 50% more testing than other bidders Installation & Training discovers problems –Helen: consultant in charge of installation & training –Ann: skeptical nurse cross-checking computer outputs © USC-CSSE25

26. University of Southern California Center for Systems and Software Engineering Mercy Hospital Pharmacy System: Problems Dosage problems from data entry errors –10x dosage; wrong patient Cross-checking incomplete; not trusted by some doctors Heavier data-entry load –Formalizing automated procedures  more info. needed –Pharmacy info > warehouse info Helen: Should go back to old system during cleanup George:- Is old system less risky? -How do we ensure cleanup will get it right? -How much will cleanup cost? Future practice: How to anticipate, avoid similar problems? © USC-CSSE26

27. University of Southern California Center for Systems and Software Engineering Outline Definitions and context –Power to do public harm or good –ACM/IEEE Software Engineering Code of Ethics Principles and examples –Rawls’ Theory of Justice –Relation to stakeholder win-win –Case study: Mercy Hospital Integrating ethics into daily software engineering practices –ICM/VBSE/MBASE –CS 577 ethics situations © USC-CSSE27

28. University of Southern California Center for Systems and Software Engineering Mercy Hospital : Use of VBSE/MBASE/ICM Win Win Spiral Results chain –Add patient safety outcome, patient stakeholder representative –Rework-business-workflows initiative, including safety checks; add clerical-staff stakeholder Stakeholder Win Win –Patient representative: safety criteria; parallel-operation phase- in –Clerical staff: prototype GUI, including safety-check support Business Case: includes added safety costs and benefits Risk Management: assess warehouse package safety, effects of workflow changes. © USC-CSSE28

29. University of Southern California Center for Systems and Software Engineering Use of VBSE/MBASE/ICM/Win Win Spiral-II Concurrent Engineering –Concurrently address business workflows, GUI prototypes, COTS alternatives, feature prioritization, cost/schedule/benefits analysis, other risks –Prepare to pass VCR, FCR, DCR, CCD, and IOC anchor point milestone reviews Monitoring and Control: Use Balanced Scorecard to track progress with respect to plans; apply corrective actions as necessary Change as Opportunity: Look for emerging COTS pharmacy-related fulfillment systems © USC-CSSE29

30. University of Southern California Center for Systems and Software Engineering CS 577 Ethics Situations Assuming your priorities match those of other stakeholders –Users: GUI; quality factor priorities –Maintainers: programming language, reuse, documentation –Customers/Owners: legacy compatibility, advanced vs. mature technology, full business case Favoring stakeholders who agree with you –Excessive privacy protection: customers vs. users Weighting stakeholders equally on each issue –Users on GUI; owners on legacy compatibility: developers on cost/schedule/risk Promising more than you can deliver Borrowing from other projects without credit Suppressing or delaying bad news © USC-CSSE30

31. University of Southern California Center for Systems and Software Engineering Conclusions Software engineers have increasing power to do public harm or good Value-based codes of ethics are hard to integrate with value-neutral software engineering practices Rawls’ Theory of Justice enables constructive approach for integrating ethics into daily software engineering practice –Stakeholder win-win with least-advantaged system dependents as success-critical stakeholders –ICM/VBSE/MBASE provides daily-practice framework, way to build trust © USC-CSSE31