1. University of Southern California Center for Software Engineering C S E USC Barry Boehm, USC CS 510 Lecture Fall 2011 Value-Based Software Engineering: Motivation and Key Practices August 26, 20111©USC-CSSE

2. University of Southern California Center for Software Engineering C S E USC August 26, 20112 Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

3. University of Southern California Center for Software Engineering C S E USC August 26, 2011 3 Software Testing Business Case Vendor proposition –Our test data generator will cut your test costs in half –We’ll provide it to you for 30% of your test costs –After you run all your tests for 50% of your original cost, you are 20% ahead Any concerns with vendor proposition? ©USC-CSSE

4. University of Southern California Center for Software Engineering C S E USC August 26, 2011 4 Software Testing Business Case Vendor proposition –Our test data generator will cut your test costs in half –We’ll provide it to you for 30% of your test costs –After you run all your tests for 50% of your original cost, you are 20% ahead Any concerns with vendor proposition? –34 reasons in 2004 ABB experience paper Unrepresentative test coverage, too much output data, lack of test validity criteria, poor test design, instability due to rapid feature changes, lack of preparation and experience (automated chaos yields faster chaos), … C. Persson and N. Yilmazturk, Proceedings, ASE 2004 –But one more significant reason ©USC-CSSE

5. University of Southern California Center for Software Engineering C S E USC August 26, 20115 Software Testing Business Case Vendor proposition –Our test data generator will cut your test costs in half –We’ll provide it to you for 30% of your test costs –After you run all your tests for 50% of your original cost, you are 20% ahead Any concerns with vendor proposition? –Test data generator is value-neutral* –Assumes every test case, defect is equally important –Usually, 20% of test cases cover 80% of business case * As are most current software engineering techniques ©USC-CSSE

6. University of Southern California Center for Software Engineering C S E USC August 26, 20116 20% of Features Provide 80% of Value: Focus Testing on These (Bullock, 2000) % of Value for Correct Customer Billing Customer Type 100 80 60 40 20 51015 Automated test generation tool - all tests have equal value ©USC-CSSE

7. University of Southern California Center for Software Engineering C S E USC August 26, 20117 Value-Based Testing Provides More Net Value Net Value NV 60 40 20 0 -20 20401006080 -40 (100, 20) Percent of tests run Test Data Generator Value-Based Testing (30, 58) % TestsTest Data GeneratorValue-Based Testing CostValueNVCostValueNV 0300-30000 103510-25105040 204020-20207555 304530-15308858 405040-10409454 …. 10080100+20100 0 ©USC-CSSE

8. University of Southern California Center for Software Engineering C S E USC August 26, 20118 By NumberP-value% Gr A higherBy ImpactP-value% Gr A higher Average of Concerns 0.20234 Average Impact of Concerns 0.04965 Average of Problems 0.05651 Average Impact of Problems 0.01289 Average of Concerns per hour 0.02655 Average Cost Effectiveness of Concerns 0.004105 Average of Problems per hour 0.02361 Average Cost Effectiveness of Problems 0.007108 Group A: 15 IV&V personnel using VBR procedures and checklists Group B 13 IV&V personnel using previous value-neutral checklists – Significantly higher numbers of trivial typo and grammar faults Value-Based Reading (VBR) Experiment — Keun Lee, ISESE 2005 ©USC-CSSE

9. University of Southern California Center for Software Engineering C S E USC August 26, 20119 Motivation for Value-Based SE Current SE methods are basically value-neutral –Every requirement, use case, object, test case, and defect is equally important –Object oriented development is a logic exercise –“Earned Value” Systems don’t track business value –Separation of concerns: SE’s job is to turn requirements into verified code –Ethical concerns separated from daily practices Value – neutral SE methods are increasingly risky –Software decisions increasingly drive system value –Corporate adaptability to change achieved via software decisions –System value-domain problems are the chief sources of software project failures ©USC-CSSE

10. University of Southern California Center for Software Engineering C S E USC August 26, 201110 Why Software Projects Fail ©USC-CSSE

11. University of Southern California Center for Software Engineering C S E USC August 26, 201111 The “Separation of Concerns” Legacy “The notion of ‘user’ cannot be precisely defined, and therefore has no place in CS or SE.” - Edsger Dijkstra, ICSE 4, 1979 “Analysis and allocation of the system requirements is not the responsibility of the SE group but is a prerequisite for their work” - Mark Paulk at al., SEI Software CMM* v.1.1, 1993 *Capability Maturity Model ©USC-CSSE

12. University of Southern California Center for Software Engineering C S E USC August 26, 201112 Resulting Project Social Structure SOFTWARE MGMT. AERO.ELEC.G & C MFG. COMMPAYLOAD I wonder when they'll give us our requirements? ©USC-CSSE

13. University of Southern California Center for Software Engineering C S E USC August 26, 201113 20% of Fires Cause 80% of Property Loss: Focus Fire Dispatching on These? 100 80 60 40 20 % of Property Loss % of Fires 20406080100 ©USC-CSSE

14. University of Southern California Center for Software Engineering C S E USC August 26, 201114 Missing Stakeholder Concerns: Fire Dispatching System Dispatch to minimize value of property loss –Neglect safety, least-advantaged property owners English-only dispatcher service –Neglect least-advantaged immigrants Minimal recordkeeping –Reduced accountability Tight budget; design for nominal case –Neglect reliability, safety, crisis performance ©USC-CSSE

15. University of Southern California Center for Software Engineering C S E USC August 26, 201115 Key Definitions Value (from Latin “valere” – to be worth 1.A fair or equivalent in goods, services, or money 2.The monetary worth of something 3.Relative worth, utility or importance Software validation (also from Latin “valere”) –Validation: Are we building the right product? –Verification: Are we building the product right? ©USC-CSSE

16. University of Southern California Center for Software Engineering C S E USC August 26, 201116 Conclusions So Far Value considerations are software success- critical “Success” is a function of key stakeholder values –Risky to exclude key stakeholders Values vary by stakeholder role Non-monetary values are important –Fairness, customer satisfaction, trust Value-based approach integrates ethics into daily software engineering practice ©USC-CSSE

17. University of Southern California Center for Software Engineering C S E USC August 26, 201117 Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

18. University of Southern California Center for Software Engineering C S E USC August 26, 201118 DMR/BRA* Results Chain INITIATIVE OUTCOME Implement a new order entry system ASSUMPTION Contribution Order to delivery time is an important buying criterion Reduce time to process order Reduced order processing cycle (intermediate outcome) Increased sales Reduce time to deliver product *DMR Consulting Group’s Benefits Realization Approach ©USC-CSSE

19. University of Southern California Center for Software Engineering C S E USC August 26, 201119 Expanded Order Processing System Benefits Chain ©USC-CSSE

20. University of Southern California Center for Software Engineering C S E USC August 26, 201120 The Model-Clash Spider Web: Master Net - Stakeholder value propositions (win conditions) ©USC-CSSE

21. University of Southern California Center for Software Engineering C S E USC August 26, 201121 Projecting Yourself Into Others’ Win Situations Do unto others.. As you would have others do unto you Counterexample: The Golden Rule ©USC-CSSE

22. University of Southern California Center for Software Engineering C S E USC August 26, 201122 Projecting Yourself Into Others’ Win Situations Build computer systems to serve users and operators.. Assuming users and operators like to write programs, and know computer science Computer sciences world (compilers, OS, etc.) –Users are programmers Applications world –Users are pilots, doctors, tellers Do unto others.. As you would have others do unto you Counterexample: The Golden Rule ©USC-CSSE

23. University of Southern California Center for Software Engineering C S E USC August 26, 201123 EasyWinWin OnLine Negotiation Steps ©USC-CSSE

24. University of Southern California Center for Software Engineering C S E USC August 26, 201124 Red cells indicate lack of consensus. Oral discussion of cell graph reveals unshared information, unnoticed assumptions, hidden issues, constraints, etc. ©USC-CSSE

25. University of Southern California Center for Software Engineering C S E USC August 26, 201125 Example of Business Case Analysis ROI= Present Value [(Benefits-Costs)/Costs] Time Return on Investment 1 2 3 Option B Option A ©USC-CSSE

26. University of Southern California Center for Software Engineering C S E USC August 26, 201126 Example of Business Case Analysis ROI= Present Value [(Benefits-Costs)/Costs] Time Return on Investment 1 2 3 Option B- Rapid Option B Option A ©USC-CSSE

27. University of Southern California Center for Software Engineering C S E USC August 26, 201127 Examples of Utility Functions: Response Time Value Time Real-Time Control; Event Support Value Time Mission Planning, Competitive Time-to-Market Value Time Event Prediction - Weather; Software Size Value Time Data Archiving Priced Quality of Service Critical Region ©USC-CSSE

28. University of Southern California Center for Software Engineering C S E USC August 26, 201128 How Much Testing is Enough? - Early Startup: Risk due to low dependability - Commercial: Risk due to low dependability - High Finance: Risk due to low dependability - Risk due to market share erosion COCOMO II: 012223454Added % test time COQUALMO: 1.0.475.24.1250.06P(L) Early Startup:.33.19.11.06.03S(L) Commercial: 1.0.56.32.18.10S(L) High Finance: 3.01.68.96.54.30S(L) Market Risk:.008.027.09.301.0RE m Sweet Spot ©USC-CSSE

29. University of Southern California Center for Software Engineering C S E USC August 26, 201129 Value-Based Defect Reduction Example: Goal-Question-Metric (GQM) Approach Goal:Our supply chain software packages have too many defects. We need to get their defect rates down Question: ? ©USC-CSSE

30. University of Southern California Center for Software Engineering C S E USC August 26, 201130 Value-Based GQM Approach – I Q: How do software defects affect system value goals? ask why initiative is needed -Order processing -Too much downtime on operations critical path -Too many defects in operational plans -Too many new-release operational problems G: New system-level goal: Decrease software-defect-related losses in operational effectiveness -With high-leverage problem areas above as specific subgoals New Q: ? ©USC-CSSE

31. University of Southern California Center for Software Engineering C S E USC August 26, 201131 Value-Based GQM Approach – II New Q: Perform system problem-area root cause analysis: ask why problems are happening via models Example: Downtime on critical path Order Validate order Produce status reports Schedule packaging, delivery Validate items in stock items Prepare delivery packages Deliver order Where are primary software-defect-related delays? Where are biggest improvement-leverage areas? –Reducing software defects in Scheduling module –Reducing non-software order-validation delays –Taking Status Reporting off the critical path –Downstream, getting a new Web-based order entry system Ask “why not?” as well as “why?” ©USC-CSSE

32. University of Southern California Center for Software Engineering C S E USC August 26, 201132 Value-Based GQM Results Defect tracking weighted by system-value priority –Focuses defect removal on highest-value effort Significantly higher effect on bottom-line business value –And on customer satisfaction levels Engages software engineers in system issues –Fits increasing system-criticality of software Strategies often helped by quantitative models –COQUALMO, iDAVE ©USC-CSSE

33. University of Southern California Center for Software Engineering C S E USC August 26, 201133 Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

34. University of Southern California Center for Software Engineering C S E USC August 26, 201134 Is This A Risk? We just started integrating the software –and we found out that COTS* products A and B just can’t talk to each other We’ve got too much tied into A and B to change Our best solution is to build wrappers around A and B to get them to talk via CORBA** This will take 3 months and $300K It will also delay integration and delivery by at least 3 months *COTS: Commercial off-the-shelf **CORBA: Common Object Request Broker Architecture ©USC-CSSE

35. University of Southern California Center for Software Engineering C S E USC August 26, 201135 Is This A Risk? We just started integrating the software –and we found out that COTS* products A and B just can’t talk to each other We’ve got too much tied into A and B to change ******* No, it is a problem –Being dealt with reactively Risks involve uncertainties –And can be dealt with pro-actively –Earlier, this problem was a risk ©USC-CSSE

36. University of Southern California Center for Software Engineering C S E USC August 26, 201136 Earlier, This Problem Was A Risk A and B are our strongest COTS choices –But there is some chance that they can’t talk to each other –Probability of loss P(L) If we commit to using A and B –And we find out in integration that they can’t talk to each other –We’ll add more cost and delay delivery by at least 3 months –Size of loss S(L) We have a risk exposure of RE = P(L) * S(L) ©USC-CSSE

37. University of Southern California Center for Software Engineering C S E USC August 26, 201137 How Can Risk Management Help You Deal With Risks? Buying information Risk avoidance Risk transfer Risk reduction Risk acceptance ©USC-CSSE

38. University of Southern California Center for Software Engineering C S E USC August 26, 201138 Risk Management Strategies: - Buying Information Let’s spend $30K and 2 weeks prototyping the integration of A and B’s HCI’s This will buy information on the magnitude of P(L) and S(L) If RE = P(L) * S(L) is small, we’ll accept and monitor the risk If RE is large, we’ll use one/some of the other strategies ©USC-CSSE

39. University of Southern California Center for Software Engineering C S E USC August 26, 201139 Other Risk Management Strategies Risk Avoidance –COTS product C is almost as good as B, and its HCI is compatible with A –Delivering on time is worth more to the customer than the small performance loss Risk Transfer –If the customer insists on using A and B, have them establish a risk reserve. –To be used to the extent that A and B have incompatible HCI’s to reconcile Risk Reduction –If we build the wrapper right now, we add cost but minimize the schedule delay Risk Acceptance –If we can solve the A and B HCI compatibility problem, we’ll have a big competitive edge on the future procurements –Let’s do this on our own money, and patent the solution ©USC-CSSE

40. University of Southern California Center for Software Engineering C S E USC August 26, 201140 Is Risk Management Fundamentally Negative? It usually is, but it shouldn’t be As illustrated in the Risk Acceptance strategy, it is equivalent to Opportunity Management Opportunity Exposure OE = P(Gain) * S(Gain) = Expected Value Buying information and the other Risk Strategies have their Opportunity counterparts –P(Gain): Are we likely to get there before the competition? –S(Gain): How big is the market for the solution? ©USC-CSSE

41. University of Southern California Center for Software Engineering C S E USC August 26, 201141 What Else Can Risk Management Help You Do? Determine “How much is enough?” for your products and processes –Functionality, documentation, prototyping, COTS evaluation, architecting, testing, formal methods, agility, discipline, … –What’s the risk exposure of doing too much? –What’s the risk exposure of doing too little? Tailor and adapt your life cycle processes –Determine what to do next (specify, prototype, COTS evaluation, business case analysis) –Determine how much of it is enough –Examples: Risk-driven spiral model and extensions (win-win, anchor points, RUP, MBASE, CeBASE Method) Get help from higher management –Organize management reviews around top-10 risks ©USC-CSSE

42. University of Southern California Center for Software Engineering C S E USC August 26, 201142 Example Large-System Risk Analysis: How Much Architecting is Enough? Large system involves subcontracting to over a dozen software/hardware specialty suppliers Early procurement package means early start –But later delays due to inadequate architecture –And resulting integration rework delays Developing thorough architecture specs reduces rework delays –But increases subcontractor startup delays ©USC-CSSE

43. University of Southern California Center for Software Engineering C S E USC August 26, 201143 How Soon to Define Subcontractor Interfaces? Risk exposure RE = Prob(Loss) * Size(Loss) -Loss due to rework delays Time spent defining & validating architecture RE = P(L) * S(L) Many interface defects: high P(L) Critical IF defects: high S(L) Few IF defects: low P(L) Minor IF defects: low S(L) ©USC-CSSE

44. University of Southern California Center for Software Engineering C S E USC August 26, 201144 How Soon to Define Subcontractor Interfaces? - Loss due to rework delays - Loss due to late subcontact startups Time spent defining & validating architecture RE = P(L) * S(L) Few delays: low P(L) Short Delays: low S(L) Many delays: high P(L) Long delays: high S(L) Many interface defects: high P(L) Critical IF defects: high S(L) Few IF defects: low P(L) Minor IF defects: low S(L) ©USC-CSSE

45. University of Southern California Center for Software Engineering C S E USC August 26, 201145 Time spent defining & validating architecture RE = P(L) * S(L) Many delays: high P(L) Long delays: high S(L) Sweet Spot Many interface defects: high P(L) Critical IF defects: high S(L) Few IF defects: low P(L) Minor IF defects: low S(L) How Soon to Define Subcontractor Interfaces? - Sum of Risk Exposures Few delays: low P(L) Short delays: low S(L) ©USC-CSSE

46. University of Southern California Center for Software Engineering C S E USC August 26, 201146 How Soon to Define Subcontractor Interfaces? -Very Many Subcontractors RE = P(L) * S(L) Higher P(L), S(L): many more IF’s Mainstream Sweet Spot Many-Subs Sweet Spot Time spent defining & validating architecture ©USC-CSSE

47. University of Southern California Center for Software Engineering C S E USC August 26, 201147 How Much Architecting Is Enough? -A COCOMO II Analysis Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule 10000 KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward ©USC-CSSE

48. University of Southern California Center for Software Engineering C S E USC August 26, 201148 Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

49. University of Southern California Center for Software Engineering C S E USC August 26, 201149 Sequential Engineering Neglects Risk $100M $50M Arch. A: Custom many cache processors Arch. B: Modified Client-Server 12 3 4 5 Response Time (sec) Original Spec After Prototyping ©USC-CSSE

50. University of Southern California Center for Software Engineering C S E USC August 26, 201150 Change As Opportunity: Agile Methods Continuous customer interaction Short value - adding increments Tacit interpersonal knowledge –Stories, Planning game, pair programming –Explicit documented knowledge expensive to change Simple design and refactoring –Vs. Big Design Up Front ©USC-CSSE

51. University of Southern California Center for Software Engineering C S E USC August 26, 201151 Five Critical Decision Factors Represent five dimensions Size, Criticality, Dynamism, Personnel, Culture Personnel Dynamism (% Requirements-change/month) Culture (% thriving on chaos vs. order) Size (# of personnel) Criticality (Loss due to impact of defects) 50 30 10 5 1 90 70 50 30 10 3 30 100 300 35 30 25 20 15 Essential Funds Discretionary Funds Comfort Single Life Many Lives (% Level 1B)(% Level 2&3) 0 10 20 30 40 Agile Plan-driven ©USC-CSSE

52. University of Southern California Center for Software Engineering C S E USC August 26, 201152 Conclusions Marketplace trends favor transition to VBSE paradigm –Software a/the major source of product value –Software the primary enabler of adaptability VBSE involves 7 key elements 1.Benefits Realization Analysis 2.Stakeholders’ Value Proposition Elicitation and Reconciliation 3.Business Case Analysis 4.Continuous Risk and Opportunity Management 5.Concurrent System and Software Engineering 6.Value-Based Monitoring and Control 7.Change as Opportunity Processes for implementing VBSE emerging –ICM, Lean Development, DMR/BRA, Balanced Scorecard, Quality Economics, Agile Methods ©USC-CSSE

53. University of Southern California Center for Software Engineering C S E USC August 26, 201153 References C. Baldwin & K. Clark, Design Rules: The Power of Modularity, MIT Press, 1999. B. Boehm, “Value-Based Software Engineering,” ACM Software Engineering Notes, March 2003. B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, 2000. B. Boehm and L. Huang, “Value-Based Software Engineering: A Case Study, Computer, March 2003, pp. 33-41. B. Boehm & K. Sullivan, “Software Economics: A Roadmap,” The Future of Software Economics, A. Finkelstein (ed.), ACM Press, 2000. B. Boehm and R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison Wesley, 2003 (to appear). J. Bullock, “Calculating the Value of Testing,” Software Testing and Quality Engineering, May/June 2000, pp. 56-62. S. Faulk, D. Harmon, and D. Raffo, “Value-Based Software Engineering (VBSE): A Value-Driven Approach to Product-Line Engineering,” Proceedings, First Intl. Conf. On SW Product Line Engineering, August 2000. ©USC-CSSE

54. University of Southern California Center for Software Engineering C S E USC August 26, 201154 R. Kaplan & D. Norton, The Balanced Scorecard: Translating Strategy into Action, Harvard Business School Press, 1996. C. Persson and N. Yilmazturk, “Establishment of Automated Regression Testing at ABB,” Proceedings, ASE 2004, August 2004, pp. 112-121. D. Reifer, Making the Software Business Case, Addison Wesley, 2002. K. Sullivan, Y. Cai, B. Hallen, and W. Griswold, “The Structure and Value of Modularity in Software Design,” Proceedings, ESEC/FSE, 2001, ACM Press, pp. 99-108. J. Thorp and DMR, The Information Paradox, McGraw Hill, 1998. Economics-Driven Software Engineering Research (EDSER) web site: www.edser.org MBASE web site : sunset.usc.edu/research/MBASE ©USC-CSSE