University of Southern California Center for Software Engineering C S E USC Barry Boehm, USC CS 510 Lecture Fall 2011 Value-Based Software Engineering: Motivation and Key Practices August 26, 20111©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Software Testing Business Case Vendor proposition –Our test data generator will cut your test costs in half –We’ll provide it to you for 30% of your test costs –After you run all your tests for 50% of your original cost, you are 20% ahead Any concerns with vendor proposition? ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Software Testing Business Case Vendor proposition –Our test data generator will cut your test costs in half –We’ll provide it to you for 30% of your test costs –After you run all your tests for 50% of your original cost, you are 20% ahead Any concerns with vendor proposition? –34 reasons in 2004 ABB experience paper Unrepresentative test coverage, too much output data, lack of test validity criteria, poor test design, instability due to rapid feature changes, lack of preparation and experience (automated chaos yields faster chaos), … C. Persson and N. Yilmazturk, Proceedings, ASE 2004 –But one more significant reason ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Software Testing Business Case Vendor proposition –Our test data generator will cut your test costs in half –We’ll provide it to you for 30% of your test costs –After you run all your tests for 50% of your original cost, you are 20% ahead Any concerns with vendor proposition? –Test data generator is value-neutral* –Assumes every test case, defect is equally important –Usually, 20% of test cases cover 80% of business case * As are most current software engineering techniques ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, % of Features Provide 80% of Value: Focus Testing on These (Bullock, 2000) % of Value for Correct Customer Billing Customer Type Automated test generation tool - all tests have equal value ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Value-Based Testing Provides More Net Value Net Value NV (100, 20) Percent of tests run Test Data Generator Value-Based Testing (30, 58) % TestsTest Data GeneratorValue-Based Testing CostValueNVCostValueNV … ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, By NumberP-value% Gr A higherBy ImpactP-value% Gr A higher Average of Concerns Average Impact of Concerns Average of Problems Average Impact of Problems Average of Concerns per hour Average Cost Effectiveness of Concerns Average of Problems per hour Average Cost Effectiveness of Problems Group A: 15 IV&V personnel using VBR procedures and checklists Group B 13 IV&V personnel using previous value-neutral checklists – Significantly higher numbers of trivial typo and grammar faults Value-Based Reading (VBR) Experiment — Keun Lee, ISESE 2005 ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Motivation for Value-Based SE Current SE methods are basically value-neutral –Every requirement, use case, object, test case, and defect is equally important –Object oriented development is a logic exercise –“Earned Value” Systems don’t track business value –Separation of concerns: SE’s job is to turn requirements into verified code –Ethical concerns separated from daily practices Value – neutral SE methods are increasingly risky –Software decisions increasingly drive system value –Corporate adaptability to change achieved via software decisions –System value-domain problems are the chief sources of software project failures ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Why Software Projects Fail ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, The “Separation of Concerns” Legacy “The notion of ‘user’ cannot be precisely defined, and therefore has no place in CS or SE.” - Edsger Dijkstra, ICSE 4, 1979 “Analysis and allocation of the system requirements is not the responsibility of the SE group but is a prerequisite for their work” - Mark Paulk at al., SEI Software CMM* v.1.1, 1993 *Capability Maturity Model ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Resulting Project Social Structure SOFTWARE MGMT. AERO.ELEC.G & C MFG. COMMPAYLOAD I wonder when they'll give us our requirements? ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, % of Fires Cause 80% of Property Loss: Focus Fire Dispatching on These? % of Property Loss % of Fires ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Missing Stakeholder Concerns: Fire Dispatching System Dispatch to minimize value of property loss –Neglect safety, least-advantaged property owners English-only dispatcher service –Neglect least-advantaged immigrants Minimal recordkeeping –Reduced accountability Tight budget; design for nominal case –Neglect reliability, safety, crisis performance ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Key Definitions Value (from Latin “valere” – to be worth 1.A fair or equivalent in goods, services, or money 2.The monetary worth of something 3.Relative worth, utility or importance Software validation (also from Latin “valere”) –Validation: Are we building the right product? –Verification: Are we building the product right? ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Conclusions So Far Value considerations are software success- critical “Success” is a function of key stakeholder values –Risky to exclude key stakeholders Values vary by stakeholder role Non-monetary values are important –Fairness, customer satisfaction, trust Value-based approach integrates ethics into daily software engineering practice ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, DMR/BRA* Results Chain INITIATIVE OUTCOME Implement a new order entry system ASSUMPTION Contribution Order to delivery time is an important buying criterion Reduce time to process order Reduced order processing cycle (intermediate outcome) Increased sales Reduce time to deliver product *DMR Consulting Group’s Benefits Realization Approach ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Expanded Order Processing System Benefits Chain ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, The Model-Clash Spider Web: Master Net - Stakeholder value propositions (win conditions) ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Projecting Yourself Into Others’ Win Situations Do unto others.. As you would have others do unto you Counterexample: The Golden Rule ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Projecting Yourself Into Others’ Win Situations Build computer systems to serve users and operators.. Assuming users and operators like to write programs, and know computer science Computer sciences world (compilers, OS, etc.) –Users are programmers Applications world –Users are pilots, doctors, tellers Do unto others.. As you would have others do unto you Counterexample: The Golden Rule ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, EasyWinWin OnLine Negotiation Steps ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Red cells indicate lack of consensus. Oral discussion of cell graph reveals unshared information, unnoticed assumptions, hidden issues, constraints, etc. ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Example of Business Case Analysis ROI= Present Value [(Benefits-Costs)/Costs] Time Return on Investment Option B Option A ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Example of Business Case Analysis ROI= Present Value [(Benefits-Costs)/Costs] Time Return on Investment Option B- Rapid Option B Option A ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Examples of Utility Functions: Response Time Value Time Real-Time Control; Event Support Value Time Mission Planning, Competitive Time-to-Market Value Time Event Prediction - Weather; Software Size Value Time Data Archiving Priced Quality of Service Critical Region ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, How Much Testing is Enough? - Early Startup: Risk due to low dependability - Commercial: Risk due to low dependability - High Finance: Risk due to low dependability - Risk due to market share erosion COCOMO II: Added % test time COQUALMO: P(L) Early Startup: S(L) Commercial: S(L) High Finance: S(L) Market Risk: RE m Sweet Spot ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Value-Based Defect Reduction Example: Goal-Question-Metric (GQM) Approach Goal:Our supply chain software packages have too many defects. We need to get their defect rates down Question: ? ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Value-Based GQM Approach – I Q: How do software defects affect system value goals? ask why initiative is needed -Order processing -Too much downtime on operations critical path -Too many defects in operational plans -Too many new-release operational problems G: New system-level goal: Decrease software-defect-related losses in operational effectiveness -With high-leverage problem areas above as specific subgoals New Q: ? ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Value-Based GQM Approach – II New Q: Perform system problem-area root cause analysis: ask why problems are happening via models Example: Downtime on critical path Order Validate order Produce status reports Schedule packaging, delivery Validate items in stock items Prepare delivery packages Deliver order Where are primary software-defect-related delays? Where are biggest improvement-leverage areas? –Reducing software defects in Scheduling module –Reducing non-software order-validation delays –Taking Status Reporting off the critical path –Downstream, getting a new Web-based order entry system Ask “why not?” as well as “why?” ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Value-Based GQM Results Defect tracking weighted by system-value priority –Focuses defect removal on highest-value effort Significantly higher effect on bottom-line business value –And on customer satisfaction levels Engages software engineers in system issues –Fits increasing system-criticality of software Strategies often helped by quantitative models –COQUALMO, iDAVE ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Is This A Risk? We just started integrating the software –and we found out that COTS* products A and B just can’t talk to each other We’ve got too much tied into A and B to change Our best solution is to build wrappers around A and B to get them to talk via CORBA** This will take 3 months and $300K It will also delay integration and delivery by at least 3 months *COTS: Commercial off-the-shelf **CORBA: Common Object Request Broker Architecture ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Is This A Risk? We just started integrating the software –and we found out that COTS* products A and B just can’t talk to each other We’ve got too much tied into A and B to change ******* No, it is a problem –Being dealt with reactively Risks involve uncertainties –And can be dealt with pro-actively –Earlier, this problem was a risk ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Earlier, This Problem Was A Risk A and B are our strongest COTS choices –But there is some chance that they can’t talk to each other –Probability of loss P(L) If we commit to using A and B –And we find out in integration that they can’t talk to each other –We’ll add more cost and delay delivery by at least 3 months –Size of loss S(L) We have a risk exposure of RE = P(L) * S(L) ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, How Can Risk Management Help You Deal With Risks? Buying information Risk avoidance Risk transfer Risk reduction Risk acceptance ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Risk Management Strategies: - Buying Information Let’s spend $30K and 2 weeks prototyping the integration of A and B’s HCI’s This will buy information on the magnitude of P(L) and S(L) If RE = P(L) * S(L) is small, we’ll accept and monitor the risk If RE is large, we’ll use one/some of the other strategies ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Other Risk Management Strategies Risk Avoidance –COTS product C is almost as good as B, and its HCI is compatible with A –Delivering on time is worth more to the customer than the small performance loss Risk Transfer –If the customer insists on using A and B, have them establish a risk reserve. –To be used to the extent that A and B have incompatible HCI’s to reconcile Risk Reduction –If we build the wrapper right now, we add cost but minimize the schedule delay Risk Acceptance –If we can solve the A and B HCI compatibility problem, we’ll have a big competitive edge on the future procurements –Let’s do this on our own money, and patent the solution ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Is Risk Management Fundamentally Negative? It usually is, but it shouldn’t be As illustrated in the Risk Acceptance strategy, it is equivalent to Opportunity Management Opportunity Exposure OE = P(Gain) * S(Gain) = Expected Value Buying information and the other Risk Strategies have their Opportunity counterparts –P(Gain): Are we likely to get there before the competition? –S(Gain): How big is the market for the solution? ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, What Else Can Risk Management Help You Do? Determine “How much is enough?” for your products and processes –Functionality, documentation, prototyping, COTS evaluation, architecting, testing, formal methods, agility, discipline, … –What’s the risk exposure of doing too much? –What’s the risk exposure of doing too little? Tailor and adapt your life cycle processes –Determine what to do next (specify, prototype, COTS evaluation, business case analysis) –Determine how much of it is enough –Examples: Risk-driven spiral model and extensions (win-win, anchor points, RUP, MBASE, CeBASE Method) Get help from higher management –Organize management reviews around top-10 risks ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Example Large-System Risk Analysis: How Much Architecting is Enough? Large system involves subcontracting to over a dozen software/hardware specialty suppliers Early procurement package means early start –But later delays due to inadequate architecture –And resulting integration rework delays Developing thorough architecture specs reduces rework delays –But increases subcontractor startup delays ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, How Soon to Define Subcontractor Interfaces? Risk exposure RE = Prob(Loss) * Size(Loss) -Loss due to rework delays Time spent defining & validating architecture RE = P(L) * S(L) Many interface defects: high P(L) Critical IF defects: high S(L) Few IF defects: low P(L) Minor IF defects: low S(L) ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, How Soon to Define Subcontractor Interfaces? - Loss due to rework delays - Loss due to late subcontact startups Time spent defining & validating architecture RE = P(L) * S(L) Few delays: low P(L) Short Delays: low S(L) Many delays: high P(L) Long delays: high S(L) Many interface defects: high P(L) Critical IF defects: high S(L) Few IF defects: low P(L) Minor IF defects: low S(L) ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Time spent defining & validating architecture RE = P(L) * S(L) Many delays: high P(L) Long delays: high S(L) Sweet Spot Many interface defects: high P(L) Critical IF defects: high S(L) Few IF defects: low P(L) Minor IF defects: low S(L) How Soon to Define Subcontractor Interfaces? - Sum of Risk Exposures Few delays: low P(L) Short delays: low S(L) ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, How Soon to Define Subcontractor Interfaces? -Very Many Subcontractors RE = P(L) * S(L) Higher P(L), S(L): many more IF’s Mainstream Sweet Spot Many-Subs Sweet Spot Time spent defining & validating architecture ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, How Much Architecting Is Enough? -A COCOMO II Analysis Percent of Project Schedule Devoted to Initial Architecture and Risk Resolution Added Schedule Devoted to Rework (COCOMO II RESL factor) Total % Added Schedule KSLOC 100 KSLOC 10 KSLOC Sweet Spot Sweet Spot Drivers: Rapid Change: leftward High Assurance: rightward ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Outline Value-based software engineering (VBSE) motivation, examples, and definitions VBSE key practices –Benefits realization analysis –Stakeholder Win-Win negotiation –Business case analysis –Continuous risk and opportunity management –Concurrent system and software engineering –Value-based monitoring and control –Change as opportunity Conclusions and references ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Sequential Engineering Neglects Risk $100M $50M Arch. A: Custom many cache processors Arch. B: Modified Client-Server Response Time (sec) Original Spec After Prototyping ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Change As Opportunity: Agile Methods Continuous customer interaction Short value - adding increments Tacit interpersonal knowledge –Stories, Planning game, pair programming –Explicit documented knowledge expensive to change Simple design and refactoring –Vs. Big Design Up Front ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Five Critical Decision Factors Represent five dimensions Size, Criticality, Dynamism, Personnel, Culture Personnel Dynamism (% Requirements-change/month) Culture (% thriving on chaos vs. order) Size (# of personnel) Criticality (Loss due to impact of defects) Essential Funds Discretionary Funds Comfort Single Life Many Lives (% Level 1B)(% Level 2&3) Agile Plan-driven ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, Conclusions Marketplace trends favor transition to VBSE paradigm –Software a/the major source of product value –Software the primary enabler of adaptability VBSE involves 7 key elements 1.Benefits Realization Analysis 2.Stakeholders’ Value Proposition Elicitation and Reconciliation 3.Business Case Analysis 4.Continuous Risk and Opportunity Management 5.Concurrent System and Software Engineering 6.Value-Based Monitoring and Control 7.Change as Opportunity Processes for implementing VBSE emerging –ICM, Lean Development, DMR/BRA, Balanced Scorecard, Quality Economics, Agile Methods ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, References C. Baldwin & K. Clark, Design Rules: The Power of Modularity, MIT Press, B. Boehm, “Value-Based Software Engineering,” ACM Software Engineering Notes, March B. Boehm, C. Abts, A.W. Brown, S. Chulani, B. Clark, E. Horowitz, R. Madachy, D. Reifer, and B. Steece, Software Cost Estimation with COCOMO II, Prentice Hall, B. Boehm and L. Huang, “Value-Based Software Engineering: A Case Study, Computer, March 2003, pp B. Boehm & K. Sullivan, “Software Economics: A Roadmap,” The Future of Software Economics, A. Finkelstein (ed.), ACM Press, B. Boehm and R. Turner, Balancing Agility and Discipline: A Guide for the Perplexed, Addison Wesley, 2003 (to appear). J. Bullock, “Calculating the Value of Testing,” Software Testing and Quality Engineering, May/June 2000, pp S. Faulk, D. Harmon, and D. Raffo, “Value-Based Software Engineering (VBSE): A Value-Driven Approach to Product-Line Engineering,” Proceedings, First Intl. Conf. On SW Product Line Engineering, August ©USC-CSSE

University of Southern California Center for Software Engineering C S E USC August 26, R. Kaplan & D. Norton, The Balanced Scorecard: Translating Strategy into Action, Harvard Business School Press, C. Persson and N. Yilmazturk, “Establishment of Automated Regression Testing at ABB,” Proceedings, ASE 2004, August 2004, pp D. Reifer, Making the Software Business Case, Addison Wesley, K. Sullivan, Y. Cai, B. Hallen, and W. Griswold, “The Structure and Value of Modularity in Software Design,” Proceedings, ESEC/FSE, 2001, ACM Press, pp J. Thorp and DMR, The Information Paradox, McGraw Hill, Economics-Driven Software Engineering Research (EDSER) web site: MBASE web site : sunset.usc.edu/research/MBASE ©USC-CSSE