University of Southern California Center for Systems and Software Engineering Barry Boehm and Jo Ann Lane University of Southern California Center for.

University of Southern California Center for Systems and Software Engineering Barry Boehm and Jo Ann Lane University of Southern California Center for Systems and Software Engineering http://csse.usc.edu http://csse.usc.edu (tech report 2009-500) Presented at ICSE 2009, May 19, 2009 Process and Product Architectures and Practices for Achieving both Agility and High Assurance ----Tutorial----

University of Southern California Center for Systems and Software Engineering May 19, 2009 Goals of Tutorial Describe –Current and new challenges with respect to the development of software-intensive systems –How the Incremental Commitment Model (ICM) and its capabilities can help developers meet these challenges –Use of risk patterns to determine which system elements can be done in an agile fashion and which should use more formal processes Review several case studies that illustrate challenges and the power of the ICM process decision table Conduct exercise to develop ICM special case model 2Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Tutorial Outline Challenges for developing next- generation software systems Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM ICM tailoring exercise 3Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Future Process Challenges-I Multi-owner, multi-mission systems of systems (SoS) –Integrated supply chain: strategic planning, marketing, merchandising, outsourcing, just-in-time manufacturing, logistics, finance, customer relations management –Over 50 separately evolving external systems or services –Need to satisfice among multiple stakeholders –No one-size-fits-all solutions or processes Emergence and human-intensiveness –Requirements not pre-specifiable –Budgets and schedules not pre-specifiable –Need for evolutionary growth –Need to manage uncertainty and risk 4Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The Broadening Early Cone of Uncertainty (CU) Need greater investments in narrowing CU –Mission, investment, legacy analysis –Competitive prototyping –Concurrent engineering –Associated estimation methods and management metrics Larger systems will often have subsystems with narrower CU’s Global Interactive, Brownfield Batch, Greenfield Local Interactive, Some Legacy 6Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Current System Acquisition Methods Easy to misinterpret as one-size-fits-all V-Model 1 Spiral Model 2 High level guidance assumes that acquirers have extensive acquisition experience... Without experience, too easy to misinterpret and auger in with disastrous results... 1 http://en.wikipedia.org/wiki/V-Model 2 http://en.wikipedia.org/wiki/Spiral_modelhttp://en.wikipedia.org/wiki/V-Modelhttp://en.wikipedia.org/wiki/Spiral_model 7Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Future Process Challenges-II Rapid pace of change –In competition, mission priorities, technology, Commercial Off-the-Shelf (COTS), environment –Need incremental development to avoid obsolescence –Need concurrent vs. sequential processes –Need both prescience and rapid adaptability Software important; humans more important Brownfield vs. Greenfield development –Need to provide legacy continuity of service –Need to accommodate legacy, OTS constraints Always-on, never-fail systems –Need well-controlled, high-assurance processes –Need to synchronize and stabilize concurrency –Need to balance assurance and agility 8Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009©USC-CSSE 9 15 July 2008 Rapid Change Creates a Late Cone of Uncertainty – Need incremental vs. one-shot development Uncertainties in competition, technology, organizations, mission priorities 9Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Tutorial Outline Challenges for developing next-generation software systems Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM ICM tailoring exercise 10Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 What is the ICM? Risk-driven framework for determining and evolving best-fit system life-cycle process Integrates the strengths of phased and risk- driven spiral process models Synthesizes together principles critical to successful system development –Commitment and accountability of system sponsors –Success-critical stakeholder satisficing –Incremental growth of system definition and stakeholder commitment –Concurrent engineering –Iterative development cycles –Risk-based activity levels and anchor point milestones Principles trump diagrams… Principles used by 60-80% of CrossTalk Top-5 projects, 2002-2005 11Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 ICM Nature and Origins Integrates hardware, software, and human factors elements of systems engineering –Concurrent exploration of needs and opportunities –Concurrent engineering of hardware, software, human aspects –Concurrency stabilized via anchor point milestones Developed in response to systems/software issues –Clarify “spiral development” usage in DoD Instruction 5000.2 Initial phased version (2005) –Explain Future Combat System of systems spiral usage to GAO Underlying process principles (2006) –Provide framework for human-systems integration National Research Council report (2007) Integrates strengths of current process models –But not their weaknesses ©USC-CSSE12Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 12/31/2007 ©USC-CSSE 13 Incremental Commitment in Gambling Total Commitment: Roulette –Put your chips on a number E.g., a value of a key performance parameter –Wait and see if you win or lose Incremental Commitment: Poker, Blackjack –Put some chips in –See your cards, some of others’ cards –Decide whether, how much to commit to proceed 13Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 12/31/2007 ©USC-CSSE 15 Total vs. Incremental Commitment – 4:1 RPV Total Commitment –Agent technology demo and PR: Can do 4:1 for $1B –Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months –PDR: many outstanding risks, undefined interfaces –$800M, 40 months: “halfway” through integration and test –1:1 IOC after $3B, 80 months Incremental Commitment [with a number of competing teams] –$25M, 6 mo. to VCR [4]: may beat 1:2 with agent technology, but not 4:1 –$75M, 8 mo. to FCR [3]: agent technology may do 1:1; some risks –$225M, 10 mo. to DCR [2]: validated architecture, high-risk elements –$675M, 18 mo. to IOC [1]: viable 1:1 capability –1:1 IOC after $1B, 42 months 15Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The Incremental Commitment Life Cycle Process: Overview Anchor Point Milestones Synchronize, stabilize concurrency via FEDs Risk patterns determine life cycle process 16Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009©USC-CSSE Anchor Point Feasibility Evidence Descriptions Evidence provided by developer and validated by independent experts that: If the system is built to the specified architecture, it will –Satisfy the requirements: capability, interfaces, level of service, and evolution –Support the operational concept –Be buildable within the budgets and schedules in the plan –Generate a viable return on investment –Generate satisfactory outcomes for all of the success-critical stakeholders All major risks resolved or covered by risk management plans Serves as basis for stakeholders’ commitment to proceed Can be used to strengthen current schedule- or event-based reviews 18Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The Incremental Commitment Life Cycle Process: Overview Anchor Point Milestones Concurrently engr. OpCon, rqts, arch, plans, prototypes Concurrently engr. Incr.N (ops), N+1 (devel), N+2 (arch) 19Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Risk-Driven Scalable Spiral Model: Increment View Rapid Change High Assurance Short, Stabilized Development Of Increment N Increment N Transition/O&M Increment N Baseline Short Development Increments Foreseeable Change (Plan) Stable Development Increments 20Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Risk-Driven Scalable Spiral Model: Increment View Agile Rebaselining for Future Increments Short, Stabilized Development of Increment N Verification and Validation (V&V) of Increment N Deferrals ArtifactsConcerns Rapid Change High Assurance Future Increment Baselines Increment N Transition/ Operations and Maintenance Future V&V Resources Increment N Baseline Current V&V Resources Unforeseeable Change (Adapt) Short Development Increments Foreseeable Change (Plan) Stable Development Increments Continuous V&V 21Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Incremental Commitment In Systems and Life: Anchor Point Milestones Common System/Software stakeholder commitment points –Defined in concert with Government, industry organizations –Initially coordinated with Rational’s Unified Software Development Process Exploration Commitment Review (ECR) –Stakeholders’ commitment to support initial system scoping –Like dating Validation Commitment Review (VCR) –Stakeholders’ commitment to support system concept definition and investment analysis –Like going steady 22Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Incremental Commitment In Systems and Life: Anchor Point Milestones (continued) Foundations Commitment Review (FCR) –Stakeholders’ commitment to support system architecting –Like getting engaged Development Commitment Review (DCR) –Stakeholders’ commitment to support system development –Like getting married Incremental Operational Capabilities (OCs) –Stakeholders’ commitment to support operations –Like having children 23Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Example ICM Commercial Application: Symbiq Medical Infusion Pump Winner of 2006 HFES Best New Design Award Described in NRC HSI Report, Chapter 5 24Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Symbiq IV Pump ICM Process - I Exploration Phase –Stakeholder needs interviews, field observations –Initial user interface prototypes –Competitive analysis, system scoping –Commitment to proceed Valuation Phase –Feature analysis and prioritization –Display vendor option prototyping and analysis –Top-level life cycle plan, business case analysis –Safety and business risk assessment –Commitment to proceed while addressing risks 25Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Symbiq IV Pump ICM Process - II Architecting Phase –Modularity of pumping channels –Safety feature and alarms prototyping and iteration –Programmable therapy types, touchscreen analysis –Failure modes and effects analyses (FMEAs) –Prototype usage in teaching hospital –Commitment to proceed into development Development Phase –Extensive usability criteria and testing –Iterated FMEAs and safety analyses –Patient-simulator testing; adaptation to concerns –Commitment to production and business plans 26Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 ICM Summary Current processes not well matched to future challenges –Emergent, rapidly changing requirements –High assurance of scalable performance and qualities Incremental Commitment Model addresses challenges –Assurance via evidence-based milestone commitment reviews, stabilized incremental builds with concurrent V&V Evidence shortfalls treated as risks –Adaptability via concurrent agile team handling change traffic and providing evidence-based rebaselining of next-increment specifications and plans –Use of critical success factor principles: stakeholder satisficing, incremental growth, concurrent engineering, iterative development, risk- based activities and milestones Major implications for funding, contracting, career paths 30Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Risks Used in Risk Comparison Environmental risks –Technology uncertainties –Many stakeholders –Complex system-of-systems –Legacy compatibility Agility risks –Scalability –Use of simple design –Personnel turnover –Too-frequent releases –Not enough agile-capable people Plan-driven risks –Rapid change –Need for rapid results –Emergent requirements –Not enough discipline-capable people 33Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Two Examples Two agent-based systems –Small – Event Managers application –Large – National Information System for Crisis Management (NISCM) application Each example results in a development strategy based on risk analyses 34Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Event Managers Project Small startup company Diverse set of smaller agent-based planning applications This project: support for managing the infrastructure and operations of conferences and conventions Widening variety of options and interdependencies, makes an agent-based approach highly attractive 35Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 NISCM Profile Broad coalition of government agencies and critical private-sector organizations Support cross-agency and public-private sector coordination of crisis management activities Adaptive mobile network, virtual collaboration, information assurance, and information integration technology Private-sector system-of-systems integration contractor 40Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The Incremental Commitment Life Cycle Process: Overview Anchor Point Milestones Synchronize, stabilize concurrency via FEDs Risk patterns determine life cycle process 47Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The ICM as Risk-Driven Process Generator Stage I of the ICM has 3 decision nodes with 4 options/node –Culminating with incremental development in Stage II –Some options involve go-backs –Results in many possible process paths Can use ICM risk patterns to generate frequently-used processes –With confidence that they fit the situation Can generally determine this in the Exploration phase –Develop as proposed plan with risk-based evidence at VCR milestone –Adjustable in later phases 48Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The ICM Process Decision Table: Key Decision Inputs Product and project size and complexity Requirements volatility Mission criticality Nature of Non-Developmental Item (NDI)* support –Commercial, open-source, reused components Organizational and Personnel Capability * NDI Definition [ DFARS] : a) any product that is available in the commercial marketplace; b) any previously developed product in use by a U.S. agency (federal, state, or local) or a foreign government that has a mutual defense agreement with the U.S.; c) any product described in the first two points above that requires only modifications to meet requirements; d) any product that is being produced, but not yet in the commercial marketplace, that satisfies the above criteria. 50Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The ICM Process Decision Table: Key Decision Outputs Key Stage I activities: incremental definition Key Stage II activities: incremental development and operations Suggested calendar time per build, per deliverable increment 51Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Common Risk-Driven Special Cases of the ICM (Cases 1-4) Case 1: Use NDI Example: Small accounting system Size, Complexity: Size variable, complexity low Typical Change Rate/Month: Negligible Criticality: n/a NDI Support: Complete Organizational Personnel Capability: NDI-experienced (medium) Key Stage I Activities (Incremental Definition): Acquire NDI Key Stage II Activities (Incremental Development/Operations): Use NDI Time/Build: n/a Time/Increment: Vendor-driven Case 2: Agile Example: E-services Size, Complexity: Low Typical Change Rate/Month: 1-30% Criticality: Low to medium NDI Support: Good, in place Organizational Personnel Capability: Agile-ready, medium-high experience Key Stage I Activities (Incremental Definition): Skip Valuation and Architecting phases Key Stage II Activities (Incremental Development/Operations): Scrum plus agile methods of choice Time/Build: <= 1 day Time/Increment: 2-6 weeks Case 3: Architected Agile Example: Business data processing Size, Complexity: Medium Typical Change Rate/Month: 1-10 % Criticality: Medium to high NDI Support: Good, most in place Organizational Personnel Capability: Agile-ready, medium to high experience Key Stage I Activities (Incremental Definition): Combine Valuation, Architecting phases. Complete NDI preparation. Key Stage II Activities (Incremental Development/Operations): Architecture-based Scrum of Scrums Time/Build: 2-4 weeks Time/Increment: 2-6 months Case 4: Formal Methods Example: Security kernel; Safety-critical LSI chip Size, Complexity: Low Typical Change Rate/Month: 0.3% Criticality: Extra high NDI Support: None Organizational Personnel Capability: Strong formal methods experience Key Stage I Activities (Incremental Definition): Precise formal specification Key Stage II Activities (Incremental Development/Operations): Formally-based programming language; formal verification Time/Build: 1-5 days Time/Increment: 1-4 weeks 52Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Common Risk-Driven Special Cases of the ICM (Cases 5-8) Case 5: Hardware with Embedded Software Component Example: Multi-sensor control device Size, Complexity: Low Typical Change Rate/Month: 0.3 - 1 % Criticality: Medium to very high NDI Support: Good, in place Organizational Personnel Capability: Experienced, medium-high Key Stage I Activities (Incremental Definition): Concurrent hardware/software engineering. CDR-level ICM DCR Key Stage II Activities (Incremental Development/Operations): IOC development, LRIP, FRP. Concurrent version N+1 engineering Time/Build: Software 1-5 days Time/Increment: Market-driven Case 6: Indivisible IOC Example: Complete vehicle platform Size, Complexity: Medium to high Typical Change Rate/Month: 0.3 – 1% Criticality: High to very high NDI Support: Some in place Organizational Personnel Capability: Experienced, medium to high Key Stage I Activities (Incremental Definition): Determine minimum- IOC likely, conservative cost. Add deferrable software features as risk reserve Key Stage II Activities (Incremental Development/Operations): Drop deferrable features to meet conservative cost. Strong award free for features not dropped. Time/Build: Software: 2-6 weeks Time/Increment: Platform: 6-18 months Case 7: NDI-Intensive Example: Supply chain management Size, Complexity: Medium to high Typical Change Rate/Month: 0.3 – 3% Criticality: Medium to very high NDI Support: NDI-driven architecture Organizational Personnel Capability: NDI-experienced, medium to high Key Stage I Activities (Incremental Definition): Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements/architecture definition Key Stage II Activities (Incremental Development/Operations): Pro- active NDI evolution influencing, NDI upgrade synchronization Time/Build: Software: 1-4 weeks Time/Increment: Systems: 6-18 months Case 8: Hybrid Agile/Plan-Driven System Example: C4ISR system Size, Complexity: Medium to very high Typical Change Rate/Month: Mixed parts; 1-10% Criticality: Mixed parts; Medium to very high NDI Support: Mixed parts Organizational Personnel Capability: Mixed parts Key Stage I Activities (Incremental Definition): Full ICM, encapsulated agile in high change, low-medium criticality parts (Often HMI, external interfaces) Key Stage II Activities (Incremental Development/Operations): Full ICM, three-team incremental development, concurrent V&V, next- increment rebaselining Time/Build: 1-2 months Time/Increment: 9-18 months 53Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Common Risk-Driven Special Cases of the ICM (Cases 9-11) Case 9: Multi-Owner Directed System of Systems Example: Net-centric military operations Size, Complexity: Very high Typical Change Rate/Month: Mixed parts; 1-10 % Criticality: Very high NDI Support: Many NDIs, some in place Organizational Personnel Capability: Related experience, medium to high Key Stage I Activities (Incremental Definition): Full ICM; extensive multi-owner team building, negotiation Key Stage II Activities (Incremental Development/Operations): Full ICM; large ongoing system/software engineering effort Time/Build: 2-4 months Time/Increment: 18-24 months Case 10: Family of Systems Example: Medical device product line Size, Complexity: Medium to very high Typical Change Rate/Month: 1-3% Criticality: Medium to very high NDI Support: Some in place Organizational Personnel Capability: Related experience, medium to high Key Stage I Activities (Incremental Definition): Skip Valuation and Architecting phases Key Stage II Activities (Incremental Development/Operations): Scrum plus agile methods of choice Time/Build: 1-2 months Time/Increment: 9-18 months Case 11: Brownfield Example: Incremental legacy phaseout Size, Complexity: High to very high Typical Change Rate/Month: 0.3-3% Criticality: Medium-high NDI Support: NDI as legacy replacement Organizational Personnel Capability: Legacy re-engineering Key Stage I Activities (Incremental Definition): Re-engineer/refactor legacy into services Key Stage II Activities (Incremental Development/Operations): Incremental legacy phaseout Time/Build: 2-6 weeks/refactor Time/Increment: 2-6 months 54Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Common Risk-Driven Special Cases of the ICM (Cases 12a/b) Case 12a: Net-Centric Services – Community Support Example: Community services or special interest group Size, Complexity: Low to medium Typical Change Rate/Month: 0.3-3% Criticality: Low to medium NDI Support: Tailorable service elements Organizational Personnel Capability: NDI-experienced Key Stage I Activities (Incremental Definition): Filter, select, compose, tailor NDI Key Stage II Activities (Incremental Development/Operations): Evolve tailoring to meet community needs Time/Build: <= 1 day Time/Increment: 2-12 months Case 12b: Net-Centric Services – Quick Response Decision Support Example: Response to competitor initiative Size, Complexity: Medium to high Typical Change Rate/Month: 3-30% Criticality: Medium to high NDI Support: Tailorable service elements Organizational Personnel Capability: NDI-experienced Key Stage I Activities (Incremental Definition): Filter, select, compose, tailor NDI Key Stage II Activities (Incremental Development/Operations): Satisfy quick response; evolve or phase out Time/Build: <= 1 day Time/Increment: Quick response-driven LEGEND C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance. CDR: Critical Design Review. DCR: Development Commitment Review. FRP: Full-Rate Production. HMI: Human-Machine Interface. HW: Hard ware. IOC: Initial Operational Capability. LSI: Large Scale Integration. LRIP: Low-Rate Initial Production. NDI: Non-Development Item. SW: Software 55Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Tutorial Outline Motivation Overview of ICM Risk-based balance of agility and assurance ICM process decision table Guidance and examples for using the ICM process decision table ICM tailoring exercise 56Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 3: Architected Agile Exploration phase determines –Need to accommodate fairly rapid change, emergent requirements, early user capability –Low risk of scalability up to 100 people –NDI support of growth envelope –Nucleus of highly agile-capable personnel –Moderate to high loss due to increment defects Example: Business data processing Size/complexity: Medium Anticipated change rate (% per month): 1-10% Criticality: Medium to high NDI support: Good, most in place Organizational and personnel capability: Agile-ready, med-high capability Key Stage I activities: Combined Valuation and Architecting phase, complete NDI preparation Key Stage II activities: Architecture-based scrum of scrums Time/build: 2-4 weeksTime/increment: 2-6 months 57Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 USA Medical Case Study 1400 software people; 7M SLOC; 7 sites –4 in Europe, 2 in India 500 medical applications; 500 financial; others Survivability-critical software problems –Reliability, productivity, performance, interoperability –Sarbanes-Oxley requirements –Management receptive to radical change Some limited experimental use of agile methods –Led by top software technologist/manager Committed to total change around Scrum and XP 58Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 USA Medical Adoption Profile July 2004 - July 2005 –Recruit top people from all sites into core team(s) –Get external expert help –Develop architecture –Early Scrum successes with infrastructure –Revise policies and practices –Train, reculture everyone –Manage expectations July 2005 – July 2006 –Begin full-scale development –Core teams as mentors 59Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Architected Agile – USA Medical Include customers and marketers –New roles; do’s/don’ts/opportunities; CRACK personnel; full collaboration and teamwork; expectations management Scrum; most XP practices; added company practices –6-12 person teams with team rooms, dedicated servers –Hourly smoke test; nightly build and regression test –Just-in-time analysis; story-point estimates; fail fast; detailed short-term plans; company architecture compliance –Embrace change in applications and practices –Global teams: wikis, daily virtual meetings, act as if next-door Release management –2-12 week architecting Sprint Zero; 3-10 1-month Sprints; Release Sprint; 1-6 month beta test –Next Sprint Zero concurrent with Release Sprint Initiative manager and team –Define practices; evolve infrastructure; provide training; guide implementation; evaluate compliance/usage; continuous improvement 60Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 7: NDI-Intensive Biggest risks: incompatible NDI; rapid change, business/mission criticality; low NDI assessment and integration experience; supply chain stakeholder incompatibilities Example: Supply chain management Size/complexity: Medium to high Anticipated change rate (% per month): 0.3-3% Criticality: Medium to very high NDI support: NDI-driven architecture Organizational and personnel capability: NDI-experienced; medium to high capability Key Stage I activities: Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements and architecture definition Key Stage II activities: Pro-active NDI evolution influencing, NDI upgrade synchronization Time/build: 1-4 weeks (software) Time/increment: 6-18 months (systems) 61Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering Example CBA: Oversize Image Viewer Client: –A librarian at USC Developers: –A development team of 5 graduates at USC Main tasks –Development of a viewing capability for oversized images, including features such as Image navigation, cataloging, search, archive, and access administration Application of CBA decision framework –First three ICM phases –Sequencing of tasks driven by stakeholders’ win conditions and projects major risk items May 19, 200964Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering ICM Application to Oversize Image Viewer Product Elaboration UseER Mapper for image navigation, display Use Mr SID for image navigation, MySQL for catalog support, Java for admin/GUI support Develop production Mr SID/My SQL/Java glue code Process Elaboration Tailor ER Mapper for library-user Windows client Prepare totailor Mr SID, My SQL to support all 3 platforms Use Schedule as Independent Variable (SAIV) process to ensure acceptable IOC in 24 weeks Stakeholder Review Customer: want campus-wide usage, support of Unix, Mac platforms ER Mapper runs only on Windows Need to address Mr SID/My SQL/Java interoperability, glue code issues; GUI usability issues Need to prioritize desired capabilities to support SAIV process Commitment Customer will find Unix, Mac user community representatives Customer will buy Mr SID Users will support GUI prototype evaluations Customer will commit to post-deployment support of software Users will commit to support training, installation, operations May 19, 200965Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering Assessing COTS’s for image viewing capabilities Use of the CBA Process Framework: Exploration New objectives May 19, 200966Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering Assessing COTS’s for additional objectives Assessing COTS’s for other features Use of the CBA Process Framework: Valuation May 19, 200968Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering Detailed assessment; tailoring and glue coding to help assessing interoperability Use of the CBA Process Framework: Foundations May 19, 200969Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 11: Brownfield Example: Incremental legacy phaseout Size/complexity: High to very high Anticipated change rate (% per month): 0.3-3 Criticality: Medium-high NDI support: NDI as legacy replacement Organizational and personnel capability: Legacy re-engineering Key Stage I activities: Re-engineer/refactor legacy into services Key Stage II activities: Incremental legacy phaseout Time/build: 2-6 week/refactor Time/increment: 2-6 months 70Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 ICM and Brownfield Development Many process models are Greenfield- oriented –Requirements→Design→Develop→Test→Operate Failed Greenfield project example –Corporate central financial system –To replace spaghetti-code collection of COBOL programs Improved ICM Brownfield approach –Concurrent new-system definition and legacy system re-engineering 71Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Failed Greenfield Corporate Financial System Used waterfall approach –Gathered requirements –Chose best-fit ERP system –Provided remaining enhancements Needed to ensure continuity of service –Planned incremental phase-in of new services Failed due to inability to selectively phase out legacy services –Dropped after 2 failed tries at cost of $40M 72Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Budgeting Legacy Systems Patched, Highly Coupled Financial and Non-Financial Services Legacy Business Services Contract Services Project Services Deliverables Management Billing Staffing Work Breakdown Structure Subcontracting Scheduling Progress Tracking Change Tracking Reqs, Configuration Management Earned Value Management 73Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 ICM Approach to Brownfield Engineering Understanding needs –Analysis of legacy system difficulties Envisioning opportunities –Concurrently decouple legacy financial and non-financial services, explore new system phase-in and architecture options System scoping and architecting –Extract legacy financial, non-financial services –Prioritize, plan for incremental financial services phase-in/out Feasibility evidence development –Successful examples of representative service extractions –Evidence of cost, schedule, performance feasibility 74Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Legacy Business Services Contract ServicesProject Services Result of Legacy Re-engineering Contract Financial Services Billing Subcontract payments... Contract Non- Financial Services Deliverables mgmt. Terms compliance... General Financial Services Accounting Budgeting Earned value Payroll... General Non- Financial Services Progress tracking Change tracking... Project Financial Services WBS Expenditure categories... Project Non- Financial Services Scheduling Staffing Reqs CM... 75Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Frequently Asked Question Q: Having all that ICM generality and then using the decision table to come back to a simple model seems like an overkill. –If my risk patterns are stable, can’t I just use the special case indicated by the decision table? A: Yes, you can and should – as long as your risk patterns stay stable. But as you encounter change, the ICM helps you adapt to it. –And it helps you collaborate with other organizations that may use different special cases. 76Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 ICM Tailoring Exercise: SupplyChain.com agent-based system 1.Analyze project and determine key decision drivers 2.Based on key decision drivers, identify candidate ICM special case for development 3.Evaluate environment, agile, and plan-driven risks, and associated risk mitigation strategies 78Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 SupplyChain.com Profile Turnkey agent-based supply chain management systems Distributed, multi-organization teams of about 50 mostly-experienced people Parts of applications are relatively stable, while others are highly volatile Architectures are driven by a few key COTS packages that are also continually evolving 79Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 The ICM Process Decision Table: Key Decision Inputs Product and project size and complexity Requirements volatility Mission criticality Nature of Non-Developmental Item (NDI)* support –Commercial, open-source, reused components Organizational and Personnel Capability * NDI Definition [ DFARS] : a) any product that is available in the commercial marketplace; b) any previously developed product in use by a U.S. agency (federal, state, or local) or a foreign government that has a mutual defense agreement with the U.S.; c) any product described in the first two points above that requires only modifications to meet requirements; d) any product that is being produced, but not yet in the commercial marketplace, that satisfies the above criteria. 80Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Common Risk-Driven Special Cases of the ICM (Cases 1-4) Case 1: Use NDI Example: Small accounting system Size, Complexity: Size variable, complexity low Typical Change Rate/Month: Negligible Criticality: n/a NDI Support: Complete Organizational Personnel Capability: NDI-experienced (medium) Key Stage I Activities (Incremental Definition): Acquire NDI Key Stage II Activities (Incremental Development/Operations): Use NDI Time/Build: n/a Time/Increment: Vendor-driven Case 2: Agile Example: E-services Size, Complexity: Low Typical Change Rate/Month: 1-30% Criticality: Low to medium NDI Support: Good, in place Organizational Personnel Capability: Agile-ready, medium-high experience Key Stage I Activities (Incremental Definition): Skip Valuation and Architecting phases Key Stage II Activities (Incremental Development/Operations): Scrum plus agile methods of choice Time/Build: <= 1 day Time/Increment: 2-6 weeks Case 3: Architected Agile Example: Business data processing Size, Complexity: Medium Typical Change Rate/Month: 1-10 % Criticality: Medium to high NDI Support: Good, most in place Organizational Personnel Capability: Agile-ready, medium to high experience Key Stage I Activities (Incremental Definition): Combine Valuation, Architecting phases. Complete NDI preparation. Key Stage II Activities (Incremental Development/Operations): Architecture-based Scrum of Scrums Time/Build: 2-4 weeks Time/Increment: 2-6 months Case 4: Formal Methods Example: Security kernel; Safety-critical LSI chip Size, Complexity: Low Typical Change Rate/Month: 0.3% Criticality: Extra high NDI Support: None Organizational Personnel Capability: Strong formal methods experience Key Stage I Activities (Incremental Definition): Precise formal specification Key Stage II Activities (Incremental Development/Operations): Formally-based programming language; formal verification Time/Build: 1-5 days Time/Increment: 1-4 weeks 81Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Risks Used in Risk Comparison Environmental risks –Technology uncertainties –Many stakeholders –Complex system-of-systems –Legacy compatibility Agility risks –Scalability –Use of simple design –Personnel turnover –Too-frequent releases –Not enough agile-capable people Plan-driven risks –Rapid change –Need for rapid results –Emergent requirements –Not enough discipline-capable people 82Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009Copyright © USC-CSSE87 ICM Summary Current processes not well matched to future challenges –Emergent, rapidly changing requirements –High assurance of scalable performance and qualities Incremental Commitment Model addresses challenges –Assurance via evidence-based milestone commitment reviews, stabilized incremental builds with concurrent V&V Evidence shortfalls treated as risks –Adaptability via concurrent agile team handling change traffic and providing evidence-based rebaselining of next-increment specifications and plans –Use of critical success factor principles: stakeholder satisficing, incremental growth, concurrent engineering, iterative development, risk- based activities and milestones –Can be adopted incrementally Major implications for funding, contracting, career paths

University of Southern California Center for Systems and Software Engineering May 19, 2009Copyright © USC-CSSE88 Implications for funding, contracting, career paths Incremental vs. total funding –Often with evidence-based competitive downselect No one-size-fits all contracting –Separate instruments for build-to-spec, agile rebaselining, V&V teams With funding and award fees for collaboration, risk management Compatible regulations, specifications, and standards Compatible acquisition corps education and training –Generally, schedule/cost/quality as independent variable Prioritized feature set as dependent variable Multiple career paths –For people good at build-to-spec, agile rebaselining, V&V –For people good at all three Future program managers and chief engineers

University of Southern California Center for Systems and Software Engineering May 19, 2009Copyright © USC-CSSE89 ICM Transition Paths Existing programs may benefit from some ICM principles and practices, but not others Problem programs may find some ICM practices helpful in recovering viability Primary opportunities for incremental adoption of ICM principles and practices –Supplementing traditional requirements and design reviews with development and review of feasibility evidence –Stabilized incremental development and concurrent architecture rebaselining –Using schedule as independent variable and prioritizing features to be delivered –Continuous verification and validation –Using the process decision table See http://csse.usc.edu (tech report 2009-500) http://csse.usc.edu

University of Southern California Center for Systems and Software Engineering May 19, 2009©USC-CSSE References - I Beck, K., Extreme Programming Explained, Addison Wesley, 1999. Boehm, B., “Some Future Trends and Implications for Systems and Software Engineering Processes”, Systems Engineering 9(1), pp. 1-19, 2006. Boehm, B., Brown, W., Basili, V., and Turner, R., “Spiral Acquisition of Software-Intensive Systems of Systems, CrossTalk, Vol. 17, No. 5, pp. 4-9, 2004. Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9, 2006. Boehm, B., and Lane, J., “Using the ICM to Integrate System Acquisition, Systems Engineering, and Software Engineering,” CrossTalk, October 2007, pp. 4-9. Boehm, B., and Lane, J., “A Process Decision Table for Integrated Systems and Software Engineering,” Proceedings, CSER 2008, April 2008. Boehm, B., “Future Challenges and Rewards for Software Engineers,” DoD Software Tech News, October 2007, pp. 6-12. Boehm, B. et al., Software Cost Estimation with COCOMO II, Prentice Hall, 2000. Boehm, B., Software Engineering Economics, Prentice Hall, 2000. Carlock, P. and Fenton, R., "System of Systems (SoS) Enterprise Systems for Information-Intensive Organizations," Systems Engineering, Vol. 4, No. 4, pp. 242-26, 2001. Carlock, P., and J. Lane, “System of Systems Enterprise Systems Engineering, the Enterprise Architecture Management Framework, and System of Systems Cost Estimation”, 21st International Forum on COCOMO and Systems/Software Cost Modeling, 2006. Checkland, P., Systems Thinking, Systems Practice, Wiley, 1980 (2nd ed., 1999). Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/, 2006.http://akss.dau.mil/dag/ Department of Defense (DoD), Instruction 5000.2, Operation of the Defense Acquisition System, May 2003. Department of Defense (DoD), Systems Engineering Plan Preparation Guide, USD(AT&L), 2004. Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System Galorath, D., and Evans, M., Software Sizing, Estimation, and Risk Management, Auerbach, 2006. Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976. 90Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009©USC-CSSE References -II Highsmith, J., Adaptive Software Development, Dorset House, 2000. International Standards Organization, Information Technology Software Life Cycle Processes, ISO/IEC 12207, 1995 ISO, Systems Engineering – System Life Cycle Processes, ISO/IEC 15288, 2002. Jensen, R. “An Improved Macrolevel Software Development Resource Estimation Model,” Proceedings, ISPA 5, April 1983, pp. 88-92. Krygiel, A., Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33 Lane, J. and Boehm, B., "System of Systems Cost Estimation: Analysis of Lead System Integrator Engineering Activities", Information Resources Management Journal, Vol. 20, No. 2, pp. 23-32, 2007. Lane, J. and Valerdi, R., “Synthesizing SoS Concepts for Use in Cost Estimation”, Proceedings of IEEE Systems, Man, and Cybernetics Conference, 2005. Lientz, B., and Swanson, E.B., Software Maintenance Management, Addison Wesley, 1980. Madachy, R., Boehm, B., Lane, J., "Assessing Hybrid Incremental Processes for SISOS Development", USC CSSE Technical Report USC-CSSE-2006-623, 2006. Maier, M., “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284). Maier, M., “System and Software Architecture Reconciliation,” Systems Engineering 9 (2), 2006, pp. 146-159. Northrop, L., et al., Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute, 2006. Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New Look, National Academy Press, 2007. Putnam, L., “A General Empirical Solution to the Macro Software Sizing and Estimating Problem,” IEEE Trans SW Engr., July 1978, pp. 345-361. Rechtin, E. Systems Architecting, Prentice Hall, 1991. Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice,” OSD/USC Integrating Systems and Software Engineering Workshop, http://csse.usc.edu/events/2007/CIIForum/pages/program.html, October 2007. http://csse.usc.edu/events/2007/CIIForum/pages/program.html 91Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009©USC-CSSE List of Acronyms B/LBaselined C4ISRCommand, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance CDConcept Development CDRCritical Design Review COTSCommercial Off-the-Shelf DCRDevelopment Commitment Review DIDevelopment Increment DoDDepartment of Defense ECRExploration Commitment Review EVMSEarned Value Management System FCRFoundations Commitment Review FEDFeasibility Evidence Description FMEAFailure Modes and Effects Analysis FRPFull-Rate Production GAOGovernment Accountability Office GUIGraphical User Interface 92Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009©USC-CSSE List of Acronyms (continued) HMIHuman-Machine Interface HSIHuman-System Interface HWHardware ICMIncremental Commitment Model IOCInitial Operational Capability IRRInception Readiness Review IS&SEIntegrating Systems and Software Engineering LCOLife Cycle Objectives LRIPLow-Rate Initial Production MBASEModel-Based Architecting and Software Engineering NDINon-Developmental Item NRCNational Research Council OCOperational Capability OCROperations Commitment Review OO&DObserve, Orient and Decide OODAObserve, Orient, Decide, Act O&MOperations and Maintenance 93Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009©USC-CSSE List of Acronyms (continued) PDRPreliminary Design Review PMProgram Manager PRPublic Relations PRRProduct Release Review RUPRational Unified Process SoSSystem of Systems SoSESystem of Systems Engineering SSESystems and Software Engineering SWSoftware SwESoftware Engineering SysESystems Engineering Sys EngrSystems Engineer S&SESystems and Software Engineering USD (AT&L)Under Secretary of Defense for Acquisition, Technology, and Logistics VCRValidation Commitment Review V&VVerification and Validation WBSWork Breakdown Structure WMIWarfighter-Machine Interface 94Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Lifecycle Evolution Motivators as Identified by CSSE Sponsors and Affiliates Current lifecycle models and associated practices –Based on outdated assumptions –Don’t always address today’s system development challenges, especially as systems become more complex Need better integration of –Systems engineering –Software engineering –Human factors engineering –Specialty engineering (e.g. information assurance, safety) Need to better identify and manage critical issues and risks up front (Young memo) Additional details found in Backup Charts… 95Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 An omniscient authority pre-establishes the requirements, preferred system concept, etc. (A4.1, 4.2, 4.4) Technical solutions are mapped and verified with respect to these (A4.1, 4.2, 4.4) They and technology do not change very often (A4.2, 4.5, 6.2) –Emphasis on rigor vs. adaptability The program has stable and controllable external interfaces (A4.5) Project organization is function-hierarchical and WBS-oriented (A3.1, 3.2) All requirements are equally important (A4.2) Reviews event/product-based vs. evidence-based (A5.2) The program’s critical path can be defined up front (A6.1) Can achieve optimum readiness at minimum life-cycle cost (A6.5) –No slack for contingencies Current SysE-SwE Guidance: Outdated Assumptions Example: SysE Plan Preparation Guide Sometimes OK for hardware; generally not for software 96Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 System/Software Architecture Mismatches - Maier, 2006 System Hierarchy –Part-of relationships; no shared parts –Function-centric; single data dictionary –Interface dataflows –Static functional- physical allocation Software Hierarchy –Uses relationships; layered multi-access –Data-centric; class- object data relations –Interface protocols; concurrency challenges –Dynamic functional- physical migration 97Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Fractionated, incompatible sensor data management “Touch Football” interface definition earned value –Full earned value taken for defining interface dataflow –No earned value left for defining interface dynamics Joining/leaving network, publish-subscribe, interrupt handling, security protocols, exception handling, mode transitions –Result: all green EVMS turns red in integration Examples of Architecture Mismatches … Sensor 1 SDMS1 Sensor 2 SDMS2 Sensor 3 SDMS3 Sensor n SDMSn …… 98Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Effect of Software Underrepresentation Software risks discovered too late Slow, buggy change management Recent large project reorganization SW (WBS-based) PM Sys EngrSoftwareC4ISR SensorsNetworks SW New 99Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 ICM Integrates Strengths of Current Process Models—But not their Weaknesses V-Model: Emphasis on early verification and validation –But not ease of sequential, single-increment interpretation Spiral Model: Risk-driven activity prioritization –But not lack of well-defined in-process milestones RUP and MBASE: Concurrent engineering stabilized by anchor point milestones –But not software orientation Lean Development: Emphasis on value-adding activities –But not repeatable manufacturing orientation Agile Methods: Adaptability to unexpected change –But not software orientation, lack of scalability ©USC-CSSE100Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 1: Use NDI Exploration phase identifies NDI opportunities NDI risk/opportunity analysis indicates risks acceptable –Product growth envelope fits within NDI capability –Compatible NDI and product evolution paths –Acceptable NDI volatility, some open-source components highly volatile –Acceptable usability, dependability, interoperability –NDI available or affordable –Example: Small accounting system Size/complexity: Low Anticipated change rate (% per month): Low Criticality: Low NDI support: Complete Organizational and personnel capability: NDI-experienced Key Stage I activities: Acquire NDI Key Stage II activities: Use NDI Time/build: Driven by time to initialize/tailor NDI Time/increment: Driven by NDI upgrades 101Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 1 Elaboration Suppose you have an application for which an appropriate NDI (COTS, open source, reuse library, customer-furnished package) solution is available, and other options of developing perhaps a better version yourself or outsourcing such a development. Even if you produce a better solution (frequently not the case), you will generally incur more expense and take longer to begin to capitalize on its benefits. And you will often find that the NDI package has features that you hadn’t realized you would need, but are there when you need them. On the other hand, there are risks that may disqualify some NDIs. They may be overly complex for your needs, incompatible with your other applications, or highly volatile. See the discussion in Section 33.1 of Software Engineering Economics (Boehm, 1981) for more about the pros and cons of NDI solutions. 102Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 2: Pure Agile Methods Exploration phase determines –Low product and project size and complexity –Fixing increment defects in next increment acceptable –Existing hardware and NDI support of growth envelope –Sufficient agile-capable personnel –Need to accommodate rapid change, emergent requirements, early user capability Example: E-services Size/complexity: Low Anticipated change rate (% per month): 1-30% Criticality: Low to medium NDI support: Good; in place Organizational and personnel capability: Agile-ready, medium to high capability Key Stage I activities: Skip Valuation and Architecting phases Key Stage II activities: Scrum plus agile methods of choice Time/build: Daily Time/increment: 2-6 weeks 103Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 2 Elaboration If your project is small (less than 10 people) and its criticality involves the loss of discretionary vs. essential funds, a pure agile method such as Extreme Programming (Beck, 1999), Scrum (Schwaber, 2002), or Crystal (Cockburn, 2002) is generally best if you have relatively high-capability, agile-ready personnel. The risks of using a more formal, plan-driven approach are less adaptability to change and belated feedback on the product’s capabilities. The biggest risk is to try to develop the application all at once for several months, and then find that it is a mismatch to the users’ needs. Agile developers have found that the best way to address this risk is to organize the project into short (2-6 week) delivery increments that may be incomplete, but provide early useful capabilities. Any flaws can then be detected early and fixed in the next increment (for very high criticality applications, this would not be acceptable). On a small project it is also easy to set up a daily build and regression test structure that identifies integration problems early when they are easier to fix. Some risks of using the agile approach is that it is harder to write a contract for what is being developed; that it may sub optimize for early success by using unscalable capabilities (fourth-generation languages, running all in main memory); or high personnel turnover. See (Boehm and Turner, 2004), pp. 121-128 for an example risk analysis of an agent-based event planning system, ending with a decision to go agile. 104Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 3 Elaboration For medium-size (20-80 people), medium complexity (reasonably mature and scalable technology; largely compatible shareholders), agile methods can be scaled using an Architected Agile approach with early investment in a largely change-prescient architecture and user/developer/customer team building. For relatively stable projects (0.3-1% change/month), plan-driven methods can be used with low risk. But for higher rates of changes (1-10%/month), a more agile approach is less risky. A risk analysis of a 50-person, medium sized architecture-based agile supply chain management project is provided on pages 106-121 of (Boehm and Turner, 2004). A number of organizations in such areas as corporate infrastructure, medical, aerospace, and ERP applications have reported significant gains in adaptability and quality of the Architected Agile approach over plan-driven methods for such projects. However, others that had less capable and agile-ready people, less management and customer commitment, and less up-front architecture investment have not. (Boehm, 2007) 105Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 4: Formal Methods Biggest risks: Software/hardware does not accurately implement required algorithm precision, security, safety mechanisms, or critical timing Example: Security kernel or safety-critical LSI chip Size/complexity: Low Anticipated change rate (% per month): 0.3% Criticality: Extra high NDI support: None Organizational and personnel capability: Strong formal methods experience Key Stage I activities: Precise formal specification Key Stage II activities: Formally-based programming language; formal verification Time/build: 1-5 days Time/increment: 1-4 weeks 106Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 4 Elaboration Formal methods involve the development and verification of a precise mathematical specification of the behavior of a hardware and/of software systems; an implementation of the system in formal-semantics-based languages; and a mathematical proof that the implementation is exactly equivalent to the specification (no more; no less). Such methods are expensive relative to standard commercial practice and require scarce high-capability personnel, but are inexpensive relative to a massive product recall, expensive lawsuits, or a massive loss of valuable and confidential information. Current formal methods generally require repeating the expensive mathematical proof process whenever the specification or implementation is changed, making the approach less viable for systems with highly volatile requirements. In general, non-developmental items are not precisely specified and verified enough to be safely used in such systems. Also, formal methods have limited scalability; almost all fully-verified software systems have less than 10,000 lines of code. However, some progress is being made toward modularizing such systems so that implementations and proofs can be built up incrementally via lower-level lemmas and theorems. 107Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 5: Hardware Component with Embedded Software Biggest risks: Device recall, lawsuits, production line rework, hardware-software integration –DCR carried to Critical Design Review level –Concurrent hardware-software design Criticality makes Agile too risky –Continuous hardware-software integration Initially with simulated hardware Low risk of overrun –Low complexity, stable requirements and NDI –Little need for risk reserve –Likely single-supplier software 108Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 5: Hardware Component with Embedded Software (continued) Example: Multi-sensor control device Size/complexity: Low Anticipated change rate (% per month): 0.3-1% Criticality: Medium to very high NDI support: Good, in place Organizational and personnel capability: Experienced; medium to high capability Key Stage I activities: Concurrent hardware and software engineering; CDR-level ICM DCR Key Stage II activities: IOC Development, LRIP,FRP, concurrent version N+1 engineering Time/build: 1-5 days (software) Time/increment: Market-driven 109Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 5 Elaboration The application classes above have been mostly software-intensive. The differences in economic and risk patterns for hardware-intensive projects (Boehm and Lane, 2007) will create different risk-based special cases of the ICM. Once a project commits to a particular manufacturing approach and infrastructure, the hardware cost of change will be much higher. And the primary costs at risk are in development and manufacturing, particularly if the component is cheaper to replace than to fix or if fixes can be accomplished by software workarounds. Thus, the ICM Stage I activities of producing and validating detailed specifications and plans are much more cost-effective than for the agile cases above. They will often go beyond the usual level of detail (Critical Design Review versus evidence- based Preliminary Design Review) for an ICM Development Commitment Review (DCR), since so many of the details are likely to be major sources of rework expenditures and delay if wrong. And after Initial Operational Capability (IOC) development, the rework risks generally dictate an incremental low rate initial production phase before a full-rate production phase in Stage II. In case the component is planned to evolve to subsequent hardware-software reconfigurations, the ICM approach of having a concurrent systems engineering team developing specifications and plans for the “N+1st “ increment could be adopted. The length of these later increments will be driven by the product’s marketplace or competitive situation. 110Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 6: Indivisible IOC Biggest risk: Complexity, NDI uncertainties cause cost- schedule overrun –Similar strategies to case 5 for criticality (CDR, concurrent HW- SW design, continuous integration) –Add deferrable software features as risk reserve Adopt conservative (90% sure) cost and schedule Drop software features to meet cost and schedule Strong award fee for features not dropped –Likely multiple-supplier software makes longer (multi-weekly) builds more necessary 111Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 6: Indivisible IOC (continued) Example: Complete vehicle platform Size/complexity: Medium to high Anticipated change rate (% per month): 0.3-1% Criticality: High to very high NDI support: Some in place Organizational and personnel capability: Experienced, medium to high capability Key Stage I activities: Determine minimum-IOC likely, conservative cost; Add deferrable software features as risk reserve Key Stage II activities: Drop deferrable features to meet conservative cost; Strong award fee for features not dropped Time/build: 2-6 weeks (software) Time/increment: 6-18 months (platform) 112Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 6 Elaboration More complex hardware intensive systems, such as aircraft, may have a great deal of software that can be incrementally developed and tested, but may have an indivisible hardware IOC that must be developed as a unit before it can be safely tested (e.g., an automobile or aircraft’s braking system or full set of safety-critical vehicle controls). Relative to the cost and duration of the software increments, the indivisible hardware IOC has two significant sources of risk: –It cannot fit into smaller software increments –It cannot drop required hardware features to meet IOC schedule/cost/quality as independent variable. –The first can be addressed by synchronizing the hardware IOC with the Nth software increment (e.g., (Rechtin and Maier, 1997) osculating orbits for hardware and software). If some combination of schedule/cost/quality is truly the project IOC’s independent variable, then one does not want to commit to a most-likely combination of schedule/cost/quality, as there will be a roughly 50% chance of an overrun or quality shortfall in completing the indivisible IOC. Rather, it is better to determine a conservative IOC cost and schedule for meeting the quality objective, and use the difference between the conservative cost and schedule and the most- likely cost and schedule as a risk reserve. The best way to do this is to use the conservative cost and schedule as the IOC target, and to determine a set of desired but non-essential, easy to drop software features that could be developed within the risk reserve. Then, if the most-likely indivisible IOC capability begins to overrun, some desired software features can be dropped without missing the scheduled delivery time and cost. There should also be a strong award-fee incentive for the developers to minimize the number of features that need to be dropped. 113Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 7 Elaboration Our experiences in developing USC web-service applications between 1996 and 2004 was that they went from 28% of the application’s functionality being delivered by NDI components to 80% (Yang et al, 2005). A similar trend was identified by the 2001 Standish Report, which reported that 53% of the functionality of commercial software applications was being delivered by NDI components in 2000 (Standish, 2001). The economics of NDI-intensive systems dictates a bottom-up versus a top-down approach to system development, in which the capability envelope of the NDI determines the affordable requirements, rather than a top-down requirements-to- capability approach. A large supply-chain management system may need to choose among several NDI candidates each for such functions as inventory control, trend analysis, supplier/customer relations management, transportation planning, manufacturing control, and financial transactions; and evaluate not only the candidates’ cost/performance aspects, but also their interoperability with each other and with the corporation’s legacy infrastructure. Besides NDI assessment, other significant sources of effort can be NDI tailoring, NDI integration, and effect of NDI version volatility and obsolescence; see (Yang et al, 2005). A particular challenge in Stage II is the effect of NDI volatility and obsolescence. Surveys have indicated that commercial NDI products have a new release about every 10 months, and that old releases are supported by the vendor for about 3 releases. Some large systems have had about 120 NDI components, indicating that about 12 components will have new releases each month, and that not upgrading will leave each component unsupported in about 30 months. In such cases, a great deal of attention needs to be paid to upgrade synchronization, and to pro-active NDI evolution influencing. Some large organizations synchronize their NDI upgrades to their major re-training cycles of about 12-18 months. For additional NDI best practices, see (Wallnau et al, 2002). 114Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 8: Hybrid Agile/Plan-Driven System Biggest risks: large scale, high complexity, rapid change, mixed high/low criticality, partial NDI support, mixed personnel capability Example: C4ISR system Size/complexity: Medium to very high Anticipated change rate (% per month): Mixed parts; 1-10% Criticality: Mixed parts; medium to very high NDI support: Mixed parts Organizational and personnel capability: Mixed parts Key Stage I activities: Full ICM; encapsulated agile in high changed; low-medium criticality parts (often HMI, external interfaces) Key Stage II activities: Full ICM, three-team incremental development, concurrent V&V, next-increment rebaselining Time/build: 1-2 months Time/increment: 9-18 months 115Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 8 Elaboration Some large, user-intensive systems such as C4ISR systems, air traffic control systems, and network control systems have a mix of relatively stable, high- criticality elements (sensors and communications; key business logic) and more volatile, more moderately critical elements such as GUI displays, electronic warfare countermeasures, and interfaces with externally evolving systems. As many acquisitions of this nature have shown, it is highly risky to apply one-size- fits-all processes and incentive structures to these differing classes of elements. Instead, in Stage I, it is important to determine which system elements belong in which class along with the other functions of understanding needs, envisioning opportunities, NDI assessment, scoping the system, and determining feasible requirements and increments performed for complex systems in Stage I; to architect the system to encapsulate the volatile parts for development by agile teams; and to organize to use plan-driven development for the other elements of the overall system. In Stage II, the cycles for the agile teams are likely to be significantly shorter than those for the plan-driven teams. 116Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 9: Multi-Owner System of Systems Biggest risks: all those of Case 8 plus –Need to synchronize, integrate separately-managed, independently-evolving systems –Extremely large-scale; deep supplier hierarchies –Rapid adaptation to change extremely difficult Example: Net-centric military operations or global supply chain management Size/complexity: Very high Anticipated change rate (% per month): Mixed parts; 1-10% Criticality: Very high NDI support: Many NDIs; some in place Organizational and personnel capability: Related experience, medium to high Key Stage I activities: Full ICM; extensive multi-owner teambuilding, negotiation Key Stage II activities: Full ICM; large ongoing system/software engineering effort Time/build: 2-4 monthsTime/increment:18-24 months 117Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 9 Elaboration In this situation, your goal is to integrate a set of existing systems (or guide and evolve the integration of a set of existing systems). These systems are primarily developed, owned, and maintained by an organization other than the one that is attempting to manage and guide the set of systems as a system of systems. Because of the independence of these constituent systems, the SoS organization has little or no formal control over the processes used to maintain and evolve the constituent systems. The SoS may be an enterprise wide business SoS, with the constituents being primarily COTS products along with some legacy applications. Or it may be Department of Defense (DoD) warfighting SoS, where the constituent legacy systems are integrated to increase capabilities on the battle field. Traditional systems engineering (SE) activities are typically tailored for the SoS case to define an SoS architecture, better coordinate the activities of multiple systems in migrating to the SoS architecture, and provide synchronization points for the SoS. In (OUSD AT&L, 2008), pilot studies have shown that many DoD SoS have re-organized the traditional SE activities into a set of seven core elements: 1) translating capability objectives, 2) understanding systems and relationships, 3) assessing performance to capability objectives, 4) developing, evolving, and maintaining SoS design, 5) monitoring and assessing changes, 6) addressing new requirements and options, and 7) orchestrating updates to SoS. Further analysis shows, that these elements map fairly well to the hybrid agile/plan-driven case (Case 8) at the SoS level. What makes this case different from Case 8, is that each of the constituent systems is using their own processes, which could be any of the above cases, depending on the scope, complexity, and characteristics of the constituent system. What is key is that the Case 9 extends Case 8 to include information or participation of the constituent systems in the agile planning activities and lets the “battle rhythm” of the constituent system increments guide the SoS plan-driven and V&V activities. 118Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 10: Family of Systems Biggest risks: all those of Case 8 plus –Need to synchronize, integrate separately-managed, independently- evolving systems –Extremely large-scale; deep supplier hierarchies –Rapid adaptation to change extremely difficult Example: Medical device product line Size/complexity: Medium to very high Anticipated change rate (% per month): 1-3% Criticality: Medium to very high NDI support: Some in place Organizational and personnel capability: Related experience, medium to high capability Key Stage I activities: Full ICM; full stakeholder participation in product line scoping; strong business case Key Stage II activities: Full ICM; extra resources for first system, version control, multi-stakeholder support Time/build: 1-2 monthsTime/increment: 9-18 months 119Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 10 Elaboration Families of systems are typically a set of systems that belong to a product line and can be easily used to customize a solution for a given need. This might be a suite of medical devices or a suite of applications to customer support. This is often the set of systems developed by a vendor that become the NDI components for CASE 7 above. Again, the rigor required for the SoS case is present here. However, in this situation, the family of systems is typically owned and evolved by a single organization/vendor and presents a case where the owning organization has much more control over the evolution of the components of the family of systems, thus possibly reducing some risks and allowing the ICM process to be a little more streamlined. 120Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 11 Elaboration A Brownfield counterexample involved a major US corporation that used a Greenfield systems engineering and development approach to develop a new Central Corporate Financial System to replace a patched-together collection of strongly-coupled and poorly documented COBOL business data processing programs. The system included an early Enterprise Resource Planning (ERP) system that was well matched to the corporation's overall financial needs, but not to its detailed business processes, which included various workarounds to compensate for difficulties with the legacy software. The new system was well organized to support incremental implementation, but was dropped at a cost of $40 million after two failed tries to provide continuity of service, due primarily to the infeasibility of incrementally phasing out the legacy software and business processes compatibly with the new system's incremental capabilities. The ICM can be used to organize systems engineering and acquisition processes in ways that better support Brownfield legacy software re-engineering so that organizations can better provide continuity of services. In particular, its concurrent activities of Understanding Needs, Envisioning Opportunities, System Scoping and Architecting, Feasibility Evidence Development, and Risk/Opportunity Assessment enable projects to focus specifically on their legacy system constraints and on opportunities to deal with them. The application of the ICM to the Brownfield corporate counterexample situation would have avoided the failures of the Greenfield development. Its Understanding Needs activity would have determined the ways in which the legacy system had intertwined financial and other business services. For examples, Project Services included budgeting and scheduling, work breakdown system accounting, earned value management intertwined with requirements, version and configuration management; Contract Services included expenditure category management, billing, and receivables management intertwined with deliverables management and engineering change proposal tracking; with similar intertwining in Personnel Services and Marketing Services. The ICM's Envisioning Opportunities activity would have identified opportunities to use large-scale refactoring methods to decouple the financial and non-financial elements of these services in ways that would make it feasible for them to interoperate with a Financial Services element. Its System Scoping and Architecting activity would have repackaged the legacy software and developed the new architecture around financial services that would support incremental phaseout, and its Feasibility Evidence Development activity would have assessed any outstanding risks and covered them with risk management plans that would be tracked to ensure project success. A good example of a Brownfield methodology with several case study examples is provided in (Hopkins and Jenkins, 2008). 121Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 12a: Net-Centric Services – Community Support Example: Community services or special interest group Size/complexity: Low to medium Anticipated change rate (% per month): 0.3-3 Criticality: Low to medium NDI support: Tailorable service elements Organizational and personnel capability: NDI-experienced Key Stage I activities: Filter, select, compose, tailor NDI Key Stage II activities: Evolve tailoring to meet community needs Time/build: <= 1 day Time/increment: 2-12 months 122Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 12a Elaboration Net-Centric Services for community support tend to fall into two categories. One involves community service organizations providing needed services for child care, elder care, handicapped, homeless, or jobless people. Their information processing service needs include tracking of clients, volunteers, donors, donations, events, and provision of news and communication services. These used to require considerable programming to realize, but now can be realized by combining and tailoring NDI packages offering such services. The other category of net-centric services for community support involves special interest groups (professionals, hobbyists, committees, ethnic groups) with similar needs for tracking members, interests, subgroups, events, and provision of news and communication services. Development of such capabilities involves much more prototyping and much less documentation than is needed for more programming- oriented applications; for example, the internals of the NDI packages are generally unavailable for architectural documentation, and the resulting system capabilities are more driven by NDI capabilities than by prespecified requirements documents. A good example of how the ICM provides support for such net-centric services is provided in (Boehm and Bhuta, 2008) and the entire journal provides additional approaches and case studies of net-centric services development. 123Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 12b: Net-Centric Services – Quick Response Decision Support Example: Response to competitor initiative Size/complexity: Medium to high Anticipated change rate (% per month): 3-30 Criticality: Medium to high NDI support: Tailorable service elements Organizational and personnel capability: NDI-experienced Key Stage I activities: Filter, select, compose, tailor NDI Key Stage II activities: Satisfy quick response; evolve or phase out Time/build: <= 1 day Time/increment: Quick response-driven 124Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering May 19, 2009 Case 12b Elaboration Another form of net-centric services involves rapidly configuring a capability to analyze alternative decision options in response to a competitor's initiative. These might involve a competitor beginning to penetrate a business's home territory, and might involve the need to evaluate the relative costs and benefits of alternative strategies of expanding services, expanding service locations, or repricing products or services. NDI packages to help analyze geographical, logistical, human resources, and financial implications of alternative competitive response decisions can be rapidly configured to provide a quick-response capability for converging on a decision. Once the decision is made, the resulting capability may be phased out, or evolved into a more general and maintainable capability for similar future situations. Again, the November/December 2008 special issue of IEEE Software on "Opportunistic Software Systems Development" provides additional perspectives on this area of net-centric services development. 125Copyright © USC-CSSE

University of Southern California Center for Systems and Software Engineering Barry Boehm and Jo Ann Lane University of Southern California Center for.

Similar presentations

Presentation on theme: "University of Southern California Center for Systems and Software Engineering Barry Boehm and Jo Ann Lane University of Southern California Center for."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

University of Southern California Center for Systems and Software Engineering Barry Boehm and Jo Ann Lane University of Southern California Center for.

Similar presentations

Presentation on theme: "University of Southern California Center for Systems and Software Engineering Barry Boehm and Jo Ann Lane University of Southern California Center for."— Presentation transcript:

Similar presentations

About project

Feedback