ICSM Principle 2: Incremental Commitment and Accountability Software Management and Economics Fall 2018 Barry Boehm, USC Presented by Jim Alstad

Outline Nature of incremental commitment Failure story: Bank of America Master Net Success story: Improving Software Processes Alternative incremental commitment processes Master Net was total commitment There are lots of ways to incrementally commit: + Build something, then wait + Gather win conditions & resolve before proceeding 8/23/2017 Copyright © USC-CSSE

Nature of Incremental Commitment Successful system development projects are built on a bedrock of trust Trust is built through effective commitments Nature of commitments Premature commitments are risky The Cones of Uncertainty Best to commit incrementally The ICSM commitment stages Incremental definition and development Without trust People don’t share information People don’t help each other Contractual overhead and bureaucracy  blame others if something goes wrong | this behavior is often what gets projects into trouble 8/23/2017 Copyright © USC-CSSE

Nature of Commitments Watts Humphrey, 1989 The person making the commitment does so willingly. The commitment is not made lightly; that is, the work involved, the resources, and the schedule are carefully considered. There is agreement between the parties as to what is to be done, by whom, and when. The commitment is openly and publicly stated. The person responsible tries to meet the commitment, even if help is needed. Prior to the committed date, if something changes that impacts either party relative to the commitment, advance notice is given and a new commitment is negotiated. Effective commitments To be accountable, important to neither promise nor expect more than can be delivered Comparison to 577a Example: Use of OSS, w/ new release 8/23/2017 Copyright © USC-CSSE

The Cones of Uncertainty – Need incremental definition and development Uncertainties in competition, technology, organizations, mission priorities World of rapid changes, very risky bet to fully commit to full details of a product to be delivered 5 years later. Black cone is basically uncertainties within the project; red cone is basically uncertainties from the external environment Early in lifecycle, many possible solutions to consider, leading to wide range of costs. Tempting to make lower bids and proposals, to win the contract. Optimistic assumptions: “Team will perform” “COTS will do everything” “Shouldn’t take that much effort to turn prototypes into product” Actual costs outrun optimistic estimates -> large overrun. Less optimistic, still risky to do fixed-price contract early. 8/23/2017 Copyright © USC-CSSE

ICSM Stages and Phases Stage I: Incremental system definition Exploration Phase: Concurrently identifies and clarifies system capability needs, constraints, and candidate solution options Valuation Phase: Analyzes alternative solutions and identifies preferred alternative Foundations Phase: Develops management and technical foundations for the selected alternative Stage II: Incremental system development Development Phase: Procurement, iterative detailed design and development, integration, and test of current-increment  components Production and Operations Phase: System “units” produced, integrated, and put into operations 8/23/2017 Copyright © USC-CSSE

The Incremental Commitment Spiral Process: Phased View Anchor Point Milestones Synchronize, stabilize concurrency via FEDs The Incremental Commitment Life Cycle Process: Overview This slide shows how the ICSM spans the life cycle process from concept exploration to operations. Each phase culminates with an anchor point milestone review. At each anchor point, there are 4 options, based on the assessed risk of the proposed system. Some options involve go-backs. These options result in many possible process paths. The life cycle is divided into two stages: Stage I of the ICSM (Definition) has 3 decision nodes with 4 options/node, culminating with incremental development in Stage II (Development and Operations). Stage II has an additional 2 decision nodes, again with 4 options/node. One can use ICSM risk patterns to generate frequently-used processes with confidence that they fit the situation. Initial risk patterns can generally be determined in the Exploration phase. One then proceeds with development as a proposed plan with risk-based evidence at the VCR milestone, adjusting in later phases as necessary. For complex systems, a result of the Exploration phase would be the Prototyping and Competition Plan discussed above. Risks associated with the system drive the life cycle process. Information about the risk(s) (feasibility assessments) supports the decision to proceed, adjust scope or priorities, or cancel the program. Risk patterns determine life cycle process 8/23/2017 Copyright © USC-CSSE 7

Outline Nature of incremental commitment Failure story: Bank of America Master Net Success story: Improving Software Processes Alternative incremental commitment processes 8/23/2017 Copyright © USC-CSSE

Failure Story: Bank of America Master Net B of A Trust Management System (TMS) B of A an early leader in banking automation Electronic check processing, 1950s-1960s Lost automation leadership by late 1970s CEO agenda to “leapfrog into 1990s” Tried $6M in-house next-generation TMS development Extensive next-generation public relations push Results: missing capabilities; lack of scalability Embarrassing with respect to public relations push CEO appointed new TMS department head Modernize or discontinue TMS business New Vice President for TMS = Trust Management System 8/23/2017 Copyright © USC-CSSE

B of A TMS Second Try: Master Net Develop full-capability requirements Union of customer wish lists: 3.5 million lines of code $22 million budget; 9 month full operational capability Look for best external TMS developer Premier Systems: several successful small-bank TMSs Lack of B of A system engineers to analyze proposals Contracted with Premier Systems March 1984 $22 million; full commitment to deliver by December 1984 December 1984: 100 programmers on-board Far from complete, even for demonstrations Many key stakeholder win condition conflicts VP – Voice of Customer – interviewed representatives of various banks and trust management organizations that will use the services Which services to provide Large set of requirements More than 100 types of assets Comply with several federal and state laws Previous in-house effort failed, so external company with successful TMS dev record – reuse Premier Systems – even if can’t reuse, capabilities in 9 months and $22M $22M / 3.5MLOC = $6 per LOC; actual costs were $50. BoA couldn’t perform detailed analysis of proposal. Lack of options, so went ahead December 1984: reused components did not scale 8/23/2017 Copyright © USC-CSSE

B of A/Premier TMS Development, 1985-88 Serious corporate image problem if discontinued Added budget, developed 1985-86 schedule Organized major 1986 demonstration of working capabilities Could not demonstrate performance scalability Still far from complete; many performance problems by end 1986 1987: Full-commitment TMS cutover to Master Net Serious performance, reliability problems, system crashes Premier Prime computer vs. BofA IBM interoperability problems Clients dropping off: 800->700 accounts; $38B->$34B assets May 1988: Transfer of TMS business to other banks Final 50-month project schedule, $80 million cost BoA management did not want to announce failure, so agreed to increase budget and prolong schedule Demo went moderately well Transfer, underpower Upgraded processors, still could not process workload Added 2 more processors, crashes continued, reports were 2 months late BoA began losing clients Original schedule: 9 months and $22M 8/23/2017 Copyright © USC-CSSE

ICSM Principles Counterexample: Bank of America Master Net This is Figure 2-2 from ICSM text. 8/23/2017 Copyright © USC-CSSE

Principles Trump Diagrams: Master Net Stakeholder value-based guidance Overconcern with Voice of Customer: 3.5 MSLOC of rqts. No concern with maintainers, interoperators: Prime vs. IBM Incremental commitment and accountability Total commitment to infeasible budget and schedule No contract award fees or penalties for under/overruns Concurrent multidiscipline engineering No prioritization of features for incremental development No prototyping of operational scenarios and usage Evidence and risk-driven decisions No evaluation of Premier Systems scalability, performance No evidence of ability to satisfy budgets and schedules 8/23/2017 Copyright © USC-CSSE

Outline Nature of incremental commitment Failure story: Bank of America Master Net Success story: Improving Software Processes Not only improve productivity, but product quality and work life Alternative incremental commitment processes This section (Improving Software Processes at TRW) is section 2.2 in the ICSM text. 8/23/2017 Copyright © USC-CSSE

Success story: TRW-SPS, 1980-82 Software Productivity System Major 1980 TRW corporate productivity initiative Need business case and plan COCOMO-based business case led to spiral plan Use of productivity opportunity tree Combination of technical, management, personnel initiatives First actual implementation of spiral model 1980-82 Summaries of Exploration, Valuation, Foundations phases Results consistently improved productivity by factor of 2 1982 International Conference on Software Engineering Best Paper TRW: Ramo & Wooldridge were technology guys, Thompson was auto parts manufacturer. Dr Boehm laid the foundation for a couple key research trends: + COCOMO 81 + Spiral Model 8/26/2015 Copyright © USC-CSSE

TRW - SPS, Exploration Phase Note imprecise language in Objectives 8/26/2015 Copyright © USC-CSSE

Productivity Improvement Framework Staffing, Incentivizing, Teambuilding Get the Best from People Facilities, Support Services Kaizen (continuous improvement) Make Tasks More Efficient Tools and Automation Work and Oversight Streamlining Collaboration Technology Lean and Agile Methods Eliminate Tasks Productivity Improvements and Tradeoffs Task Automation Model-Based Product Generation Early Risk and Defect Elimination Eliminate Scrap, Rework Evidence-Based Decision Gates Modularity Around Sources of Change Incremental, Evolutionary Development Value-Based, Agile Process Maturity Risk-Based Prototyping Simplify Products (KISS) Value-Based Capability Prioritization Satisficing vs. Optimizing Performance Reuse Components Domain Engineering and Architecture Composable Components,Services, COTS Legacy System Repurposing Reduce Operations, Support Costs Automate Operations Elements Design for Maintainability, Evolvability Value- and Architecture-Based Tradeoffs and Balancing Streamline Supply Chain Anticipate, Prepare for Change Copyright © USC-CSSE 8/26/2015

Costing Insights: COCOMO II Productivity Ranges Scale Factor Ranges: 10, 100, 1000 KSLOC Development Flexibility (FLEX) Staffing Team Cohesion (TEAM) Develop for Reuse (RUSE) Teambuilding Precedentedness (PREC) Architecture and Risk Resolution (RESL) Continuous Improvement Platform Experience (PEXP) Data Base Size (DATA) Required Development Schedule (SCED) Language and Tools Experience (LTEX) Process Maturity (PMAT) Storage Constraint (STOR) Use of Software Tools (TOOL) Platform Volatility (PVOL) Applications Experience (AEXP) Multi-Site Development (SITE) Documentation Match to Life Cycle Needs (DOCU) Required Software Reliability (RELY) Personnel Continuity (PCON) Time Constraint (TIME) Programmer Capability (PCAP) Analyst Capability (ACAP) Product Complexity (CPLX) 1 1.2 1.4 1.6 1.8 2 2.2 2.4 Productivity Range Copyright © USC-CSSE 8/26/2015

TRW-SPS, Validation Phase 8/26/2015 Copyright © USC-CSSE

TRW-SPS, Foundations Phase 8/26/2015 Copyright © USC-CSSE

Outline Nature of incremental commitment Failure story: Bank of America Master Net Success story: Improving Software Processes Alternative incremental commitment processes 8/23/2017 Copyright © USC-CSSE

Incremental and Evolutionary Acquisition Models Time phasing terms: Scoping; Architecting; Developing; Producing; Operating (SADPO) Prespecified Sequential S A ; D P O1 ; D P O2 ; D P O3 ; … Evolutionary Sequential S A D P O1 ; S A D P O2 ; S A D P O3 ; … Evolutionary Overlapped Evolutionary Concurrent S A D P O1 ; S A1 ; D1 ; P O1 S A D P O2 ; S A2 ; D2 ; P O2 S A D P O2 ; S A3 ; D3 ; P O3 8/23/2017 Copyright © USC-CSSE

Incremental and Evolutionary Acquisition Models Examples, Pros, and Cons Type Examples Pros Cons Prespecified Sequential Platform base plus Pre-Planned Product Improvements Prespecifiable full-capability requirements, scalability when stable Emergent requirements or rapid change, architecture breakers Evolutionary Sequential Small: Agile Larger: Rapid fielding Adaptability to change, need for usage feedback Easiest-first; late, costly fixes; SysE time gaps Slow for large systems Evolutionary Overlapped (DoDI 5000.02) Stable development; Maturing technology Mature technology upgrades Emergent requirements or rapid change; SysE time gaps Evolutionary Concurrent (ICSM) Rapid, emergent development Systems of systems Emergent requirements or rapid change, SysE continuity Overkill on small or highly stable systems The Incremental and Evolutionary Acquisition models shown above are represented in terms of the various ways they sequence the primary system development activities: system scoping (determining the overall system boundaries), system architecting (concurrently defining the system’s build-to requirements, architecture, and life cycle plans), and system development ,production, and operation The Prespecified Sequential incremental development model (Waterfall) is not evolutionary. It just splits up the development in order to field an early Initial Operational Capability and several pre-planned product Improvements (P3Is). When requirements are prespecifiable and stable, it enables a strong, predictable process. When requirements are emergent and/or rapidly changing, it often requires expensive rework when it needs to undo architectural commitments. Multistep: Do the requirements and architecture upfront, then plan out into increments for development. The Evolutionary Sequential model rapidly develops an initial operational capability and upgrades it based on operational experience. Pure agile software development fits this model: if something is wrong, it will be fixed in 30 days in the next release. Rapid fielding also fits this model for larger or hardware-software systems. Its strength is getting quick-response capabilities in the field. For pure agile, it can fall prey to an easiest-first set of architectural commitments which break when, for example, it tries to add security as new feature in a later increment. For rapid fielding, it may be expensive to keep the SE and development teams together while waiting for usage feedback, but it may be worth it. Evolutionary Overlapped , per DoDI 5000.02, defers the next increment until the new technology is mature enough to be added, attractive opportunity becomes available, or other resources become available, such as key personnel. Synchronize upgrades of multiple COTS. Even though the activities are overlapping, development team and system engineers spend most of the time in technology-related support. Cannot provide the support, then, for rapid change. Evolutionary Concurrent has the systems engineers handling the change traffic and rebaselining the plans and specifications for the next increment, while keeping the development stabilized for the current increment. Its example and pros and cons are provided in a following chart. 8/23/2017 Copyright © USC-CSSE

Incremental Process Decision Table 8/23/2017 Copyright © USC-CSSE

Risk-Driven Scalable Spiral Model: Increment View For each level of systems-of-interest Unforeseeable Change (Adapt) Rapid Change Agile Rebaselining for Future Increments Future Increment Baselines Short Development Increments Deferrals Foreseeable Change (Plan) Short, Stabilized Development of Increment N Increment N Transition/ Operations and Maintenance Increment N Baseline Evolutionary Concurrent, ICSM -> another view Continuing team of system engineering handling change traffic, rebaseline plans and specifications for next increment Development team stabilized for on-time, high-assurance delivery of current increment Concurrent verification and validation to perform defect detection Stable Development Increments High Assurance Artifacts Concerns Future V&V Resources Current V&V Resources Verification and Validation (V&V) of Increment N Continuous V&V 8/23/2017 Copyright © USC-CSSE

Agile Change Processing and Rebaselining Assess Changes, Propose Handling Stabilized Increment-N Development Team Change Proposers Future Increment Managers Agile Future- Increment Rebaselining Team Negotiate change disposition Formulate, analyze options in context of other changes Handle Accepted changes Discuss, resolve deferrals to future increments Propose Changes Discuss, revise, defer, or drop Rebaseline future-increment Foundations packages Prepare for rebaselined development Defer some Increment-N capabilities Recommend handling in current increment Accept changes Handle in current rebaseline Proposed changes Recommend no action, provide rationale Recommend deferrals to future increments Same idea as previous slide, but how agile teams interact with change proposers To handle unpredictable, asynchronous change traffic, it is not possible to do a sequential observe, then orient, then decide process. Instead, it requires handling unforeseen change requests involving new technology or COTS opportunities; changing requirements and priorities; changing external interfaces; low-priority current increment features being deferred to a later increment; and user requests based on experience with currently-fielded increments (including defect fixes). In the context of the agile rebaselining team in Chart 10, this chart shows how the agile team interacts with the change proposers, current-increment development team, and managers of future increments to evaluate proposed changes and their interactions with each other, and to negotiate rebaselined Milestone B packages for the next increment. Again, there is no precise way to forecast the budget and schedule of this team’s workload. Within an available budget and schedule, the agile team will perform a continuing “handle now; incorporate in next increment rebaseline; defer-or-drop” triage in the top “Assess Changes, Propose Handling” box. Surges in demand would have to be met by surges in needed expertise and funding support. 8/23/2017 Copyright © USC-CSSE

ICSM as Risk-Driven Process Generator Stage I of the ICSM has 3 decision nodes with 4 options/node Culminating with incremental development in Stage II Some options involve go-backs Results in many possible process paths Can use ICSM risk patterns to generate frequently-used processes With confidence that they fit the situation Can generally determine this in the Exploration phase Develop as proposed plan with risk-based evidence at VCR milestone Adjustable in later phases 8/23/2017 Copyright © USC-CSSE

Different Risk Patterns Yield Different Processes As illustrated in the four example paths through the Incremental Commitment Model in this slide, the ICSM is not a single monolithic one-size-fits-all process model. As with the spiral model, it is a risk-driven process model generator, but the ICSM makes it easier to visualize how different risks create different processes. In Example A, a simple business application based on an appropriately-selected Enterprise Resource Planning (ERP) package, there is no need for a Valuation or Foundations activity if there is no risk that the ERP package and its architecture will not cost-effectively support the application. Thus, one could go directly into the Development phase, using an agile method such as a Scrum/Extreme Programming combination would be a good fit. There is no need for Big Design Up Front (BDUF) activities or artifacts because an appropriate architecture is already present in the ERP package. Nor is there a need for heavyweight waterfall or V-model specifications and document reviews. The fact that the risk at the end of the Exploration phase is negligible implies that sufficient risk resolution of the ERP package’s human interface has been done. Example B involves the upgrade of several incompatible legacy applications into a service-oriented web-based system. Here, one could use a sequential waterfall or V-model if the upgrade requirements were stable, and its risks were low. However, if for example the legacy applications’ user interfaces were incompatible with each other and with web-based operations, a concurrent risk-driven spiral, waterfall, or V-model that develops and exercise extensive user interface prototypes and generates a Feasibility Evidence Description (described on chart 12) would be preferable. In Example C, the stakeholders may have found during the Valuation phase that their original assumptions about the stakeholders having a clear, shared vision and compatible goals with respect the proposed new system’s concept of operation and its operational roles and responsibilities were optimistic. In such a case, it is better to go back and assure stakeholder value proposition compatibility and feasibility before proceeding, as indicated by the arrow back into the valuation phase. In Example D, it is discovered before entering the Development phase that a superior product has already entered the marketplace, leaving the current product with an infeasible business case. Here, unless a viable business case can be made by adjusting the project’s scope, it is best to discontinue it. It is worth pointing out that it is not necessary to proceed to the next major milestone before terminating a clearly non-viable project, although stakeholder concurrence in termination is essential. 8/23/2017 Copyright © USC-CSSE Copyright © USC-CSSE 28 28

The ICSM Process Decision Table: Key Decision Inputs Product and project size and complexity Requirements volatility Mission criticality Nature of Non-Developmental/COTS Item support Commercial, open-source, reused components Organizational and Personnel Capability Info to consider when deciding what process and lifecycle looks like 8/23/2017 Copyright © USC-CSSE

The ICSM Process Decision Table: Key Decision Outputs Key Stage I activities: incremental definition Key Stage II activities: incremental development and operations Suggested calendar time per build, per deliverable increment The output of the deciding what the process and lifecycle look like 8/23/2017 Copyright © USC-CSSE

Common Risk-Driven Special Cases of the ICSM (Cases 1-4) Case 1: Use NDI Example: Small accounting system Size, Complexity: Size variable, complexity low Typical Change Rate/Month: Negligible Criticality: n/a NDI Support: Complete Organizational Personnel Capability: NDI-experienced (medium) Key Stage I Activities (Incremental Definition): Acquire NDI Key Stage II Activities (Incremental Development/Operations): Use NDI Time/Build: n/a Time/Increment: Vendor-driven Case 2: Agile Example: E-services Size, Complexity: Low Typical Change Rate/Month: 1-30% Criticality: Low to medium NDI Support: Good, in place Organizational Personnel Capability: Agile-ready, medium-high experience Key Stage I Activities (Incremental Definition): Skip Valuation and Architecting phases Key Stage II Activities (Incremental Development/Operations): Scrum plus agile methods of choice Time/Build: <= 1 day Time/Increment: 2-6 weeks Case 3: Architected Agile Example: Business data processing Size, Complexity: Medium Typical Change Rate/Month: 1-10 % Criticality: Medium to high NDI Support: Good, most in place Organizational Personnel Capability: Agile-ready, medium to high experience Key Stage I Activities (Incremental Definition): Combine Valuation, Architecting phases. Complete NDI preparation. Key Stage II Activities (Incremental Development/Operations): Architecture-based Scrum of Scrums Time/Build: 2-4 weeks Time/Increment: 2-6 months Case 4: Formal Methods Example: Security kernel; Safety-critical LSI chip Typical Change Rate/Month: 0.3% Criticality: Extra high NDI Support: None Organizational Personnel Capability: Strong formal methods experience Key Stage I Activities (Incremental Definition): Precise formal specification Key Stage II Activities (Incremental Development/Operations): Formally-based programming language; formal verification Time/Build: 1-5 days Time/Increment: 1-4 weeks 8/23/2017 Copyright © USC-CSSE

Common Risk-Driven Special Cases of the ICSM (Cases 5-8) Case 5: Hardware with Embedded Software Example: Multi-sensor control device Size, Complexity: Medium Typical Change Rate/Month: 0.3 - 1 % Criticality: Medium to very high NDI Support: Good, in place Organizational Personnel Capability: Experienced, medium-high Key Stage I Activities (Incremental Definition): Concurrent hardware/software engineering. CDR-level ICSM DCR Key Stage II Activities (Incremental Development/Operations): IOC development, LRIP, FRP. Concurrent version N+1 engineering Time/Build: Software 1-5 days Time/Increment: Market-driven Case 6: Indivisible IOC Example: Complete vehicle platform Size, Complexity: Medium to high Typical Change Rate/Month: 0.3 – 1% Criticality: High to very high NDI Support: Some in place Organizational Personnel Capability: Experienced, medium to high Key Stage I Activities (Incremental Definition): Determine minimum-IOC likely, conservative cost. Add deferrable software features as risk reserve Key Stage II Activities (Incremental Development/Operations): Drop deferrable features to meet conservative cost. Strong award free for features not dropped. Time/Build: Software: 2-6 weeks Time/Increment: Platform: 6-18 months Case 7: NDI-Intensive Example: Supply chain management Typical Change Rate/Month: 0.3 – 3% Criticality: Medium to very high NDI Support: NDI-driven architecture Organizational Personnel Capability: NDI-experienced, medium to high Key Stage I Activities (Incremental Definition): Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements/architecture definition Key Stage II Activities (Incremental Development/Operations): Pro-active NDI evolution influencing, NDI upgrade synchronization Time/Build: Software: 1-4 weeks Time/Increment: Systems: 6-18 months Case 8: Hybrid Agile/Plan-Driven System Example: C4ISR system Size, Complexity: Medium to very high Typical Change Rate/Month: Mixed parts; 1-10% Criticality: Mixed parts; Medium to very high NDI Support: Mixed parts Organizational Personnel Capability: Mixed parts Key Stage I Activities (Incremental Definition): Full ICSM, encapsulated agile in high change, low-medium criticality parts (Often HMI, external interfaces) Key Stage II Activities (Incremental Development/Operations): Full ICSM, three-team incremental development, concurrent V&V, next-increment rebaselining Time/Build: 1-2 months Time/Increment: 9-18 months 8/23/2017 Copyright © USC-CSSE

Common Risk-Driven Special Cases of the ICSM (Cases 9-11) Case 9: Multi-Owner System of Systems Example: Net-centric military operations; Metro crisis management Size, Complexity: Very high Typical Change Rate/Month: Mixed parts; 1-10 % Criticality: Very high NDI Support: Many NDIs, some in place Organizational Personnel Capability: Related experience, medium to high Key Stage I Activities (Incremental Definition): Full ICSM; extensive multi-owner team building, negotiation Key Stage II Activities (Incremental Development/Operations): Full ICSM; large ongoing system/software engineering effort Time/Build: 2-4 months Time/Increment: 18-24 months Case 10: Family of Systems Example: Medical device product line Size, Complexity: Medium to very high Typical Change Rate/Month: 1-3% Criticality: Medium to very high NDI Support: Some in place Key Stage I Activities (Incremental Definition): Skip Valuation and Architecting phases Key Stage II Activities (Incremental Development/Operations): Scrum plus agile methods of choice Time/Build: 1-2 months Time/Increment: 9-18 months Case 11: Brownfield Example: Incremental legacy phaseout Size, Complexity: High to very high Typical Change Rate/Month: 0.3-3% Criticality: Medium-high NDI Support: NDI as legacy replacement Organizational Personnel Capability: Legacy re-engineering Key Stage I Activities (Incremental Definition): Re-engineer/refactor legacy into services Key Stage II Activities (Incremental Development/Operations): Incremental legacy phaseout Time/Build: 2-6 weeks/refactor Time/Increment: 2-6 months 8/23/2017 Copyright © USC-CSSE