University of Southern California Center for Systems and Software Engineering Barry Boehm, Jo Ann Lane, Supannika Koolmanojwong, Richard Turner USC-CSSE ARR, April 29, 2014 Life Cycle Cost Modeling and Risk Assessment for 21 st Century Enterprises

University of Southern California Center for Systems and Software Engineering Outline Challenges for 21 st Century Enterprises Incremental Commitment Spiral Model (ICSM) Framework for Addressing Challenges Proposed Approach for Cost Modeling Updated Survey of Top-10 Risks 1/13/2014© USC-CSSE2

University of Southern California Center for Systems and Software Engineering Challenges for 21 st Century Enterprises Need to deal continually with the 3 D’s –Dependability: 24/7 assurance of capabilities Requires high assurance of robust systems, SoSs –Dynamism: coping with unforeseen change Requires versatility, modifiability, adaptability Dependability + Dynamism = Resilience –Diversity: of D + D needs across enterprise No one-size-fits-all solutions Need to identify enterprise frameworks for generating system and SoS solutions –ICSM provides process framework for doing this Need corresponding capabilities for cost, schedule, and tradespace modeling 1/13/2014© USC-CSSE3

University of Southern California Center for Systems and Software Engineering ICSM Framework for Addressing 3D Challenges Identify success-critical stakeholders (SCSHs) and their 3D value propositions –At enterprise-portfolio level and portfolio-project level Explore alternative enterprise and portfolio architectures and their ability to support value propositions at all three levels –Develop, evaluate evidence of feasibility for each –Baseline most satisfactory combination Pilot, evaluate, evolve baseline combination –Identify risks of further use for each –If risks unacceptable to SCSHs, descope or discontinue And explore alternative approaches to satisfying needs Avoids Procrustean Bed of one-size-fits-all solutions 1/13/2014© USC-CSSE4

University of Southern California Center for Systems and Software Engineering The Procrustean Bed Procrustes: Greek Mythology –Rogue smith and bandit –Hostel with one-size-fits-all bed –Guests too small: stretch them to fit –Guests too large: lop off the offending parts 1/13/2014© USC-CSSE5

University of Southern California Center for Systems and Software Engineering Build Your Own Procrustean Bed Pure Waterfall, Vee: Fixed Price and Spec Contract –Lop off needed changes as requirements creep Pure Agile: Easiest First; Dedicated On-Site Customer –Later scalability and assurance problems; single-failure point Voice of the Customer: Accept All “Requirements” –Gold-plating; neglect voices of acquirer, developer, owner Piling on Incompatible Constraints: No Way Out –Project Example: Waterfall, COTS, Ada, GOTS Reuse Inflexible Standards: No Choice But Tailoring Down –MIL-STD-498: choice of 23, 6, or 1 DID denied Overconstrained Maturity Models: Excluding Expertise –Software CMM: Exclude software group from system rqts. 1/13/2014© USC-CSSE6

University of Southern California Center for Systems and Software Engineering 1/13/2014 Current System Acquisition Methods Too easy to misinterpret as one-size-fits-all V-Model 1 Spiral Model 2 High level guidance assumes that acquirers have extensive acquisition experience... Without experience, too easy to misinterpret and auger in with disastrous results © USC-CSSE

University of Southern California Center for Systems and Software Engineering Procrustean Example: DoD Acquisition Process

University of Southern California Center for Systems and Software Engineering Progress: Draft DoDI MODEL 1: HARDWARE INTENSIVE PROGRAM MODEL 2: DEFENSE UNIQUE SOFTWARE INTENSIVE PROGRAM MODEL 3: INCREMENTALLY FIELDED SOFTWARE INTENSIVE PROGRAM MODEL 4: ACCELERATED ACQUISITION PROGRAM Hybrid Program A (Hardware Dominant). Hybrid Program B (Software Dominant). 1/13/2014© USC-CSSE9

University of Southern California Center for Systems and Software Engineering 1/13/2014 What is the ICSM? Risk-driven framework for determining and evolving best-fit system life-cycle process Integrates the strengths of phased and risk- driven spiral process models Synthesizes together principles critical to successful system development –Stakeholder value-based guidance –Incremental commitment and accountability –Concurrent multidiscipline engineering –Evidence and risk-driven decisions Principles trump diagrams… Principles used by 60-80% of CrossTalk Top-5 projects, © USC-CSSE

University of Southern California Center for Systems and Software Engineering The Incremental Commitment Spiral Model 1/13/201411© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 ICSM Nature and Origins Integrates hardware, software, and human factors elements of systems life cycle –Concurrent exploration of needs and opportunities –Concurrent engineering of hardware, software, human aspects –Concurrency stabilized via anchor point milestones Developed in response to a variety of issues –Clarify “spiral development” usage Initial phased version (2005) –Provide framework for human-systems integration National Research Council report (2007) Integrates strengths of current process models –But not their weaknesses Facilitates transition from existing practices –Electronic Process Guide (2009) Copyright © USC-CSSE12© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014Copyright © USC-CSSE 13 Incremental Commitment in Gambling Total Commitment: Roulette –Put your chips on a number E.g., a value of a key performance parameter –Wait and see if you win or lose Incremental Commitment: Poker, Blackjack –Put some chips in –See your cards, some of others’ cards –Decide whether, how much to commit to proceed 13© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 The Incremental Commitment Spiral Process: Phased View Anchor Point Milestones Synchronize, stabilize concurrency via FEDs Risk patterns determine life cycle process 14© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 ICSM Activity Levels for Complex Systems 15© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 Anchor Point Feasibility Evidence Descriptions Evidence provided by developer and validated by independent experts that: If the system is built to the specified architecture, it will –Satisfy the requirements: capability, interfaces, level of service, and evolution –Support the operational concept –Be buildable within the budgets and schedules in the plan –Generate a viable return on investment –Generate satisfactory outcomes for all of the success-critical stakeholders All major risks resolved or covered by risk management plans Serves as basis for stakeholders’ commitment to proceed Synchronizes and stabilizes concurrent activities Can be used to strengthen current schedule- or event-based reviews 16© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 The ICSM as Risk-Driven Process Generator Stage I of the ICSM has 3 decision nodes with 4 options/node –Culminating with incremental development in Stage II –Some options involve go-backs –Results in many possible process paths Can use ICSM risk patterns to generate frequently-used processes –With confidence that they fit the situation Can generally determine this in the Exploration phase –Develop as proposed plan with risk-based evidence at VCR milestone –Adjustable in later phases 17© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014Copyright © USC-CSSE 18 Different Risk Patterns Yield Different Processes 18© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 The ICSM Process Decision Table: Key Decision Inputs Product and project size and complexity Requirements volatility Mission criticality Nature of Non-Developmental/COTS/Services support –Commercial, open-source, reused components –Cloud services Organizational and Personnel Capability 19© USC-CSSE

University of Southern California Center for Systems and Software Engineering ICSM Common Case Decision Table

University of Southern California Center for Systems and Software Engineering 1/13/2014 Case 2: Architected Agile Exploration phase determines –Need to accommodate fairly rapid change, emergent requirements, early user capability –Low risk of scalability up to 100 people –NDI support of growth envelope –Nucleus of highly agile-capable personnel –Moderate to high loss due to increment defects Example: Business data processing Size/complexity: Medium Anticipated change rate (% per month): 1-10% Criticality: Medium to high NDI support: Good, most in place Organizational and personnel capability: Agile-ready, med-high capability Key Stage I activities: Combined Valuation and Architecting phase, complete NDI preparation Key Stage II activities: Architecture-based scrum of scrums Time/build: 2-4 weeksTime/increment: 2-6 months 21© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 USA Medical Case Study 1400 software people; 7M SLOC; 7 sites –4 in Europe, 2 in India 500 medical applications; 500 financial; others Survivability-critical software problems –Reliability, productivity, performance, interoperability –Sarbanes-Oxley requirements –Management receptive to radical change Some limited experimental use of agile methods –Led by top software technologist/manager Committed to total change around Scrum and XP 22© USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 USA Medical Adoption Profile July July 2005 –Recruit top people from all sites into core team(s) –Get external expert help –Develop architecture –Early Scrum successes with infrastructure –Revise policies and practices –Train, reculture everyone –Manage expectations July 2005 – July 2006 –Begin full-scale development –Core teams as mentors 23© USC-CSSE

University of Southern California Center for Systems and Software Engineering Architected Agile Approach Uses Scrum of Scrums approach –Up to 10 Scrum teams of 10 people each –Has worked for distributed international teams –Going to three levels generally infeasible General approach shown below –Often tailored to special circumstances 1/13/ © USC-CSSE

University of Southern California Center for Systems and Software Engineering 1/13/2014 Architected Agile – USA Medical Include customers and marketers –New roles; do’s/don’ts/opportunities; CRACK personnel; full collaboration and teamwork; expectations management Scrum; most XP practices; added company practices –6-12 person teams with team rooms, dedicated servers –Hourly smoke test; nightly build and regression test –Just-in-time analysis; story-point estimates; fail fast; detailed short-term plans; company architecture compliance –Embrace change in applications and practices –Global teams: wikis, daily virtual meetings, act as if next-door Release management –2-12 week architecting Sprint Zero; month Sprints; Release Sprint; 1-6 month beta test –Next Sprint Zero concurrent with Release Sprint Initiative manager and team –Define practices; evolve infrastructure; provide training; guide implementation; evaluate compliance/usage; continuous improvement 25© USC-CSSE

University of Southern California Center for Systems and Software Engineering 3/22/01©USC-CSE26 Approach Used to Evolve ICSM FCR success condition –Describes at least one feasible architecture –Satisfying requirements within cost/schedule/resource constraints –Viable cost-effective business case –Stakeholder concurrence on key system parameters Projects That Failed FCR Criteria -1996: 4 out of 16 (25%) -1997: 4 out of 15 (27%) why?

University of Southern California Center for Systems and Software Engineering 3/22/01©USC-CSE27 The Two Cultures: What’s Easy and Hard? Easy/hard things for software people “If you can do queries with all those ands, ors, synonyms, data ranges, etc., it should be easy to do natural language queries.” “If you can scan the document and digitize the text, it should be easy to digitize the figures too.” Easy/hard things for librarians “It was nice that you could add this access feature, but it overly (centralizes, decentralizes) control of our intellectual property rights.” “It was nice that you could extend the system to serve the medical people, but they haven’t agreed to live with our usage guidelines.”

University of Southern California Center for Systems and Software Engineering 3/22/01©USC-CSE Simplifier/Complicator Experiment Identify application simplifiers and complicators –For each digital library sub-domain –For both developers and clients Provide with explanations to developers and clients –Highlight relation to risk management Homework exercise to analyze simplifiers and complicators –For two of upcoming digital library projects Evaluate effect on LCO review failure rate

University of Southern California Center for Systems and Software Engineering 3/22/01©USC-CSE29 Example S&C’s for Developers 1, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 20, 31, 32, 35, 36, 37, 39 Type of Application Simple Block DiagramExamplesSimplifiersComplicators Multimedia Archive · Use standard query languages · Use standard or COTS search engine · Uniform media formats · Natural language processing · Automated cataloging or indexing · Digitizing large archives · Digitizing complex or fragile artifacts · Automated annotation/descrip tion/ or meanings to digital assets · Integration of legacy systems MM asset info Catalog MM Archive query MM asset update query update notification · Rapid access to large Archives · Access to heterogeneous media collections

University of Southern California Center for Systems and Software Engineering 3/22/01©USC-CSE30 The Results Projects That Failed LCO Criteria -1996: 4 out of 16 (25%) -1997: 4 out of 15 (27%) -1998: 1 out of 19 (5%) -1999: 1 out of 22 (5%) -2000: 2 out of 20 (10%) Post-1997 failures due to non-S&C causes –Team cohesion, client outages

University of Southern California Center for Systems and Software Engineering Outline Challenges for 21 st Century Enterprises Incremental Commitment Spiral Model (ICSM) Framework for Addressing Challenges Proposed Approach for Cost Modeling Updated Survey of Top-10 Risks 1/13/2014© USC-CSSE31

University of Southern California Center for Systems and Software Engineering Example: COCOMO II Cost Model 1995: one-size-fits-all model for 21 st century software 1999: poor fit for schedule-optimized projects; CORADMO 2000: poor fit for COTS-intensive projects: COCOTS 2003: need model for product line investment: COPLIMO 2003: poor fit for agile projects: Agile COCOMO II (partial) 2012: poor fit for incremental development: COINCOMO 1/13/2014© USC-CSSE32

University of Southern California Center for Systems and Software Engineering COQUALMO 1998 COCOMO COPROMO 1998 COSoSIMO 2007 Legend: Model has been calibrated with historical project data and expert (Delphi) data Model is derived from COCOMO II Model has been calibrated with expert (Delphi) data COCOTS 2000 COSYSMO 2005 CORADMO 1999,2012 iDAVE 2004 COPLIMO 2003 COPSEMO 1998 COCOMO II 2000 DBA COCOMO 2004 COINCOMO 2004,2012 COSECMO 2004 Software Cost Models Software Extensions Other Independent Estimation Models Dates indicate the time that the first paper was published for the model COTIPMO 2011 AGILE C II 2003 COCOMO Family of Cost Models 2/7/2014USC-CSSE/SERC33

University of Southern California Center for Systems and Software Engineering COCOMO II Data by 5-Year Periods 1/13/2014© USC-CSSE34

University of Southern California Center for Systems and Software Engineering COCOMO II Data: Productivity Trends 1/13/2014© USC-CSSE35

University of Southern California Center for Systems and Software Engineering COCOMO II Data: Process Maturity Trends 1/13/2014© USC-CSSE36

University of Southern California Center for Systems and Software Engineering SRDR Data: Productivity vs. Size, CMM Level 1/13/2014© USC-CSSE37

University of Southern California Center for Systems and Software Engineering Trends Confounded by Missing Variables Incremental Development Productivity Decline 12/03/2013Copyright © USC-CSSE38

University of Southern California Center for Systems and Software Engineering COCOMO II Status and COCOMO III Plans Baseline COCOMO II calibrated to 161 project data points –Pred (20%; 30%) = (63%; 70%) general; (75%;80%) local Added 149 data points –Pred (30%) < 40% general; some sources ~70-80% local –Some improvement with added variables (year; domain; agility) But some data-source mismatches unexplainable Vu Nguyen analysis of full dataset suggests further adjusting –Rating scales for experience, tools, reliability Proposed approach for COCOMO III –Explore models for unexplained existing sources or drop –Try added variables for mostly-general fit to existing data –Obtain more data to validate results 1/13/2014© USC-CSSE39

University of Southern California Center for Systems and Software Engineering COSYSMO 2.0 Status and COSYSMO 3.0 Plans Current COSYSMO 2.0 covers SysE with reuse (SEWR) –But not SysE for reuse (SEFR), requirements volatility (SERV) Also have a COSYSMO covering SERV but not SEWR, SEFR Based on Boeing data vs. BAE Systems data for COSYSMO 2.0 Proposed approach for COSYSMO 3.0 –Converge rating scales for SEWR, SEFR, SERV –Apply to existing BAE, Boeing data points –Obtain additional data points including SEWR, SEFR, SERV Including original COSYSMO 1.0 data points where possible –Derive best-fit COCOMO 3.0 for overall dataset Perhaps involving added variables, domain subsetting –Implies continuation of NDAs for full dataset Also explore COSYSMO as an improved framework for prediction of system development costs 1/13/2014© USC-CSSE40

University of Southern California Center for Systems and Software Engineering Outline Challenges for 21 st Century Enterprises Incremental Commitment Spiral Model (ICSM) Framework for Addressing Challenges Proposed Approach for Cost Modeling Updated Survey of Top-10 Risks 1/13/2014© USC-CSSE41

University of Southern California Center for Systems and Software Engineering Updated Survey of Top-10 Risks ICSM book Risk chapter has list of top-10 risks –Based on Thammanoon survey of several top-10 risk lists 1.Inflated expectations 2.Success-critical stakeholder lack of involvement or value conflicts 3.Underdefined plans and requirements 4.Architecture/reuse/non-developmental item (NDI) conflicts 5.Personnel shortfalls 6.Immature or obsolete processes; unbridled requirements volatility 7.Human–system integration shortfalls 8.Legacy asset incompatibilities 9.Unbalanced nonfunctional requirements or -ilities 10.Immature technology 1/13/2014© USC-CSSE42

University of Southern California Center for Systems and Software Engineering Candidates for Top-10 risks _______ Bad architectural decisions _______ Bad NDI (COTS, services, open source, reuse) decisions _______ Immature technology _______ Inadequate or misunderstood requirements _______ Lack of senior management commitment _______ Misaligned stakeholders _______ Neglected stakeholders _______ Obsolete processes, technology _______ Personnel shortfalls _______ Uncontrollable external interfaces _______ Unrealistic budgets and/or schedules (Inflated Expectations) _______ Unresolved quality-attribute conflicts _______ Unsatisfactory human-system interface _______ Weak/no analysis of alternatives _______ Weak business case, competition assessment _______ ________________________________________________________ 1/13/2014© USC-CSSE43