1. University of Southern California Center for Systems and Software Engineering Barry Boehm CS 510, Fall 2015 ICSM Patterns and Common Cases

2. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE2 03/19/2008 ©USC-CSSE The Incremental Commitment Life Cycle Process: Overview Stage I: DefinitionStage II: Development and Operations Anchor Point Milestones Synchronize, stabilize concurrency via FEDs Risk patterns determine life cycle process

3. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE3 The ICM as Risk-Driven Process Generator Stage I of the ICM has 3 decision nodes with 4 options/node –Culminating with incremental development in Stage II –Some options involve go-backs –Results in many possible process paths Can use ICM risk patterns to generate frequently-used processes –With confidence that they fit the situation Can generally determine this in the Exploration phase –Develop as proposed plan with risk-based evidence at VCR milestone –Adjustable in later phases

4. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE4 03/19/2008 ©USC-CSSE 4 Different Risk Patterns Yield Different Processes

5. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE5 The ICM Process Decision Table: Key Decision Inputs Product and project size and complexity Requirements volatility Mission criticality Nature of Non-Developmental Item (NDI)* support –Commercial, open-source, reused components Organizational and Personnel Capability * NDI Definition [ DFARS] : a) any product that is available in the commercial marketplace; b) any previously developed product in use by a U.S. agency (federal, state, or local) or a foreign government that has a mutual defense agreement with the U.S.; c) any product described in the first two points above that requires only modifications to meet requirements; d) any product that is being produced, but not yet in the commercial marketplace, that satisfies the above criteria.

6. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE6 The ICM Process Decision Table: Key Decision Outputs Key Stage I activities: incremental definition Key Stage II activities: incremental development and operations Suggested calendar time per build, per deliverable increment

7. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE7 Common Risk-Driven Special Cases of the ICM Special CaseExampleSize, Complexi ty Change Rate % /Month CriticalityNDI SupportOrg, Personnel Capability Key Stage I Activities : Incremental Definition Key Stage II Activities: Incremental Development, Operations Time per Build; per Increment 1. Use NDISmall Accounting CompleteAcquire NDIUse NDI 2. AgileE-servicesLow1 – 30Low- Med Good; in place Agile-ready Med-high Skip Valuation, Architecting phases Scrum plus agile methods of choice <= 1 day; 2-6 weeks 3. Architected Agile Business data processing Med1 – 10Med- High Good; most in place Agile-ready Med-high Combine Valuation, Architecting phases. Complete NDI preparation Architecture-based Scrum of Scrums 2-4 weeks; 2-6 months 4. HW component with embedded SW Multisensor control device Low0.3 – 1Med- Very High Good; In place Experienced; med-high Concurrent HW/SW engineering. CDR-level ICM DCR IOC Development, LRIP, FRP. Concurrent Version N+1 engineering SW: 1-5 days; Market-driven 5. Indivisible IOC Complete vehicle platform Med – High 0.3 – 1High- Very High Some in placeExperienced; med-high Determine minimum-IOC likely, conservative cost. Add deferrable SW features as risk reserve Drop deferrable features to meet conservative cost. Strong award fee for features not dropped SW: 2-6 weeks; Platform: 6-18 months 6. NDI- Intensive Supply Chain Management Med – High 0.3 – 3Med- Very High NDI-driven architecture NDI- experienced; Med-high Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements/ architecture definition Pro-active NDI evolution influencing, NDI upgrade synchronization SW: 1-4 weeks; System: 6-18 months 7. Hybrid agile / plan-driven system C4ISRMed – Very High Mixed parts: 1 – 10 Mixed parts; Med- Very High Mixed parts Full ICM; encapsulated agile in high change, low-medium criticality parts (Often HMI, external interfaces) Full ICM,three-team incremental development, concurrent V&V, next-increment rebaselining 1-2 months; 9-18 months 8. Multi-owner system of systems Net-centric military operations Very High Mixed parts: 1 – 10 Very High Many NDIs; some in place Related experience, med-high Full ICM; extensive multi-owner team building, negotiation Full ICM; large ongoing system/software engineering effort 2-4 months; 18-24 months 9. Family of systems Medical Device Product Line Med – Very High 1 – 3Med – Very High Some in placeRelated experience, med – high Full ICM; Full stakeholder participation in product line scoping. Strong business case Full ICM. Extra resources for first system, version control, multi-stakeholder support 1-2 months; 9-18 months 9. Multi-owner system of systems Net-centric military operations Very high Mixed parts; 1-10 Very High Many NDIs; some in place Related experience, med-high E Full ICM; extensive multi- owner team building, negotiation Full ICM: large ongoing system/software engineering effort 2-4 months; 18-24 months 10. Family of systems Medical device product line Med- Very High 1-3Med- Very High Some in placeRelated experience, med-high Full ICM; Full stakeholder participation in product line scoping. Strong business case. Full ICM. Extra resources for first system, version control, multi-stakeholder support 1-2 months; 9-18 months C4ISR: Command, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance. CDR: Critical Design Review. DCR: Development Commitment Review. FRP: Full-Rate Production. HMI: Human-Machine Interface. HW: Hard ware. IOC: Initial Operational Capability. LRIP: Low-Rate Initial Production. NDI: Non-Development Item. SW: Software

8. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE8 Case 1: Use NDI Exploration phase identifies NDI opportunities NDI risk/opportunity analysis indicates risks acceptable –Product growth envelope fits within NDI capability –Compatible NDI and product evolution paths –Acceptable NDI volatility, some open-source components highly volatile –Acceptable usability, dependability, interoperability –NDI available or affordable –Example: Small accounting system Size/complexity: Low Anticipated change rate (% per month): Low Criticality: Low NDI support: Complete Organization and personnel capability: NDI-experienced Key Stage I activities: Acquire NDI Key Stage II activities: Use NDI Time/build: Driven by time to initialize/tailor NDI Time/increment: Driven by NDI upgrades

9. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE9 Case 2: Pure Agile Methods Exploration phase determines –Low product and project size and complexity –Fixing increment defects in next increment acceptable –Existing hardware and NDI support of growth envelope –Sufficient agile-capable personnel –Need to accommodate rapid change, emergent requirements, early user capability Example: E-services Size/complexity: Low Anticipated change rate (% per month): 1-30% Criticality: Low to medium NDI support: Good; in place Organization and personnel capability: Agile-ready, medium to high capability Key Stage I activities: Skip Valuation and Architecting phases Key Stage II activities: Scrum plus agile methods of choice Time/build: Daily Time/increment: 2-6 weeks

10. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE10 Case 3: Architected Agile Exploration phase determines –Need to accommodate fairly rapid change, emergent requirements, early user capability –Low risk of scalability up to 100 people –NDI support of growth envelope –Nucleus of highly agile-capable personnel –Moderate to high loss due to increment defects Example: Business data processing Size/complexity: Medium Anticipated change rate (% per month): 1-10% Criticality: Medium to high NDI support: Good, most in place Organization and personnel capability: Agile-ready, med-high capability Key Stage I activities: Combined Valuation and Architecting phase, complete NDI preparation Key Stage II activities: Architecture-based scrum of scrums Time/build: 2-4 weeksTime/increment: 2-6 months

11. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE11 Medical Case Study -- USA 1400 software people; 7M SLOC; 7 sites –4 in Europe, 2 in India 500 medical applications; 500 financial; others Survivability-critical software problems –Reliability, productivity, performance, interoperability –Sarbanes-Oxley requirements –Management receptive to radical change Some limited experimental use of agile methods –Led by top software technologist/manager Committed to total change around Scrum and XP

12. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE12 Medical-USA Adoption Profile July 2004 - July 2005 –Recruit top people from all sites into core team(s) –Get external expert help –Develop architecture –Early Scrum successes with infrastructure –Revise policies and practices –Train, reculture everyone –Manage expectations July 2005 – July 2006 –Begin full-scale development –Core teams as mentors

13. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE13 Key Practices – USA Medical Include customers and marketers –New roles; do’s/don’ts/opportunities; CRACK personnel; full collaboration and teamwork; expectations management Scrum; most XP practices; added company practices –6-12 person teams with team rooms, dedicated servers –Hourly smoke test; nightly build and regression test –Just-in-time analysis; story-point estimates; fail fast; detailed short-term plans; company architecture compliance –Embrace change in applications and practices –Global teams: wikis, daily virtual meetings, act as if next-door Release management –2-12 week architecting Sprint Zero; 3-10 1-month Sprints; Release Sprint; 1-6 month beta test –Next Sprint Zero concurrent with Release Sprint Initiative manager and team –Define practices; evolve infrastructure; provide training; guide implementation; evaluate compliance/usage; continuous improvement

14. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE14 Case 4: Formal Methods Biggest risks: Software/hardware does not accurately implement required algorithm precision, security, safety mechanisms, or critical timing Example: Security kernel or safety-critical LSI chip Size/complexity: Low Anticipated change rate (% per month): 0.3% Criticality: Extra high NDI support: None Organization and personnel capability: Strong formal methods experience Key Stage I activities: Precise formal specification Key Stage II activities: Formally-based programming language; formal verification Time/build: 1-5 days Time/increment: 1-4 weeks

15. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE15 Case 5: Hardware Component with Embedded Software Biggest risks: Device recall, lawsuits, production line rework, hardware-software integration –DCR carried to Critical Design Review level –Concurrent hardware-software design Criticality makes Agile too risky –Continuous hardware-software integration Initially with simulated hardware Low risk of overrun –Low complexity, stable requirements and NDI –Little need for risk reserve –Likely single-supplier software

16. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE16 Case 5: Hardware Component with Embedded Software (continued) Example: Multi-sensor control device Size/complexity: Low Anticipated change rate (% per month): 0.3-1% Criticality: Medium to very high NDI support: Good, in place Organization and personnel capability: Experienced; medium to high capability Key Stage I activities: Concurrent hardware and software engineering; CDR-level ICM DCR Key Stage II activities: IOC Development, LRIP,FRP, concurrent version N+1 engineering Time/build: 1-5 days (software) Time/increment: Market-driven

17. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE17 Case 6: Indivisible IOC Biggest risk: Complexity, NDI uncertainties cause cost- schedule overrun –Similar strategies to case 5 for criticality (CDR, concurrent HW- SW design, continuous integration) –Add deferrable software features as risk reserve Adopt conservative (90% sure) cost and schedule Drop software features to meet cost and schedule Strong award fee for features not dropped –Likely multiple-supplier software makes longer (multi-weekly) builds more necessary

18. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE18 Case 6: Indivisible IOC (continued) Example: Complete vehicle platform Size/complexity: Medium to high Anticipated change rate (% per month): 0.3-1% Criticality: High to very high NDI support: Some in place Organization and personnel capability: Experienced, medium to high capability Key Stage I activities: Determine minimum-IOC likely, conservative cost; Add deferrable software features as risk reserve Key Stage II activities: Drop deferrable features to meet conservative cost; Strong award fee for features not dropped Time/build: 2-6 weeks (software) Time/increment: 6-18 months (platform)

19. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE19 Case 7: NDI-Intensive Biggest risks: incompatible NDI; rapid change, business/mission criticality; low NDI assessment and integration experience; supply chain stakeholder incompatibilities Example: Supply chain management Size/complexity: Medium to high Anticipated change rate (% per month): 0.3-3% Criticality: Medium to very high NDI support: NDI-driven architecture Organization and personnel capability: NDI-experienced; medium to high capability Key Stage I activities: Thorough NDI-suite life cycle cost-benefit analysis, selection, concurrent requirements and architecture definition Key Stage II activities: Pro-active NDI evolution influencing, NDI upgrade synchronization Time/build: 1-4 weeks (software) Time/increment: 6-18 months (systems)

20. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE20 Case 8: Hybrid Agile/Plan-Driven System Biggest risks: large scale, high complexity, rapid change, mixed high/low criticality, partial NDI support, mixed personnel capability Example: C4ISR system Size/complexity: Medium to very high Anticipated change rate (% per month): Mixed parts; 1-10% Criticality: Mixed parts; medium to very high NDI support: Mixed parts Organization and personnel capability: Mixed parts Key Stage I activities: Full ICM; encapsulated agile in high changed; low-medium criticality parts (often HMI, external interfaces) Key Stage II activities: Full ICM, three-team incremental development, concurrent V&V, next-increment rebaselining Time/build: 1-2 months Time/increment: 9-18 months

21. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE21 Case 9: Multi-Owner System of Systems Biggest risks: all those of Case 8 plus –Need to synchronize, integrate separately-managed, independently-evolving systems –Extremely large-scale; deep supplier hierarchies –Rapid adaptation to change extremely difficult Example: Net-centric military operations or global supply chain management Size/complexity: Very high Anticipated change rate (% per month): Mixed parts; 1-10% Criticality: Very high NDI support: Many NDIs; some in place Organization and personnel capability: Related experience, medium to high Key Stage I activities: Full ICM; extensive multi-owner teambuilding, negotiation Key Stage II activities: Full ICM; large ongoing system/software engineering effort Time/build: 2-4 monthsTime/increment:18-24 months

22. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE22 Case 10: Family of Systems Biggest risks: all those of Case 8 plus –Need to synchronize, integrate separately-managed, independently- evolving systems –Extremely large-scale; deep supplier hierarchies –Rapid adaptation to change extremely difficult Example: Medical device product line Size/complexity: Medium to very high Anticipated change rate (% per month): 1-3% Criticality: Medium to very high NDI support: Some in place Organization and personnel capability: Related experience, medium to high capability Key Stage I activities: Full ICM; full stakeholder participation in product line scoping; strong business case Key Stage II activities: Full ICM; extra resources for first system, version control, multi-stakeholder support Time/build: 1-2 monthsTime/increment: 9-18 months

23. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE23 ICM and Brownfield Development Many process models are Greenfield- oriented –Requirements→Design→Develop→Test→Operate Failed Greenfield project example –Corporate central financial system –To replace spaghetti-code collection of COBOL programs Improved ICM Brownfield approach –Concurrent new-system definition and legacy system re-engineering

24. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE24 Failed Greenfield Corporate Financial System Used waterfall approach –Gathered requirements –Chose best-fit ERP system –Provided remaining enhancements Needed to ensure continuity of service –Planned incremental phase-in of new services Failed due to inability to selectively phase out legacy services –Dropped after 2 failed tries at cost of $40M

25. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE25 Budgeting Legacy Systems Patched, Highly Coupled Financial and Non-Financial Services Legacy Business Services Contract Services Project Services Deliverables Management Billing Staffing Work Breakdown Structure Subcontracting Scheduling Progress Tracking Change Tracking Reqs, Configuration Management Earned Value Management

26. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE26 ICM Approach to Brownfield Engineering Understanding needs –Analysis of legacy system difficulties Envisioning opportunities –Concurrently decouple legacy financial and non-financial services, explore new system phase-in and architecture options System scoping and architecting –Extract legacy financial, non-financial services –Prioritize, plan for incremental financial services phase-in/out Feasibility evidence development –Successful examples of representative service extractions –Evidence of cost, schedule, performance feasibility

27. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE27 Legacy Business Services Contract ServicesProject Services Result of Legacy Re-engineering Contract Financial Services Billing Subcontract payments... Contract Non- Financial Services Deliverables mgmt. Terms compliance... General Financial Services Accounting Budgeting Earned value Payroll... General Non- Financial Services Progress tracking Change tracking... Project Financial Services WBS Expenditure categories... Project Non- Financial Services Scheduling Staffing Reqs CM...

28. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE28 Frequently Asked Question Q: Having all that ICM generality and then using the decision table to come back to a simple model seems like an overkill. –If my risk patterns are stable, can’t I just use the special case indicated by the decision table? A: Yes, you can and should – as long as your risk patterns stay stable. But as you encounter change, the ICM helps you adapt to it. –And it helps you collaborate with other organizations that may use different special cases.

29. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE29 Using Risk to Balance Assurance and Agility - Overview

30. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE30 Risks Used in Risk Comparison Environmental risks –Technology uncertainties –Many stakeholders –Complex system-of-systems –Legacy compatibility Agility risks –Scalability –Use of simple design –Personnel turnover –Too-frequent releases –Not enough agile-capable people Plan-driven risks –Rapid change –Need for rapid results –Emergent requirements –Not enough discipline-capable people

31. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE31 Four Examples Three agent-based systems –Medium – SupplyChain.com application –Small – Event Managers application –Large – National Information System for Crisis Management (NISCM) application Medical case Each example results in a development strategy based on risk analyses

32. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE32 SupplyChain.com Profile Turnkey agent-based supply chain management systems Distributed, multi-organization teams of about 50 people Parts of applications are relatively stable, while others are highly volatile Architectures are driven by a few key COTS packages that are also continually evolving

33. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE33 SupplyChain.com Home Ground Chart

34. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE34 SupplyChain.com Risks

35. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE35 SupplyChain.com Mitigations

36. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE36 SupplyChain.com Strategy LCA LCO

37. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE37 Event Managers Project Small startup company Diverse set of smaller agent-based planning applications This project: support for managing the infrastructure and operations of conferences and conventions Widening variety of options and interdependencies, makes an agent-based approach highly attractive

38. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE38 Event Managers Home Ground Chart

39. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE39 Event Managers Risks

40. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE40 Event Managers Mitigations

41. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE41 Event Managers Strategy LCA

42. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE42 NISCM Profile Broad coalition of government agencies and critical private-sector organizations Support cross-agency and public-private sector coordination of crisis management activities Adaptive mobile network, virtual collaboration, information assurance, and information integration technology Private-sector system-of-systems integration contractor

43. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE43 NISCM Home Ground Chart

44. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE44 NISCM Risks

45. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE45 NISCM Mitigations

46. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE46 NISCM Strategy LCALCO

47. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE47 Risk Profiles Across Agent-Based Examples

48. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE48©USC-CSSE References - I Beck, K., Extreme Programming Explained, Addison Wesley, 1999. Boehm, B., “Some Future Trends and Implications for Systems and Software Engineering Processes”, Systems Engineering 9(1), pp. 1-19, 2006. Boehm, B., Brown, W., Basili, V., and Turner, R., “Spiral Acquisition of Software-Intensive Systems of Systems, CrossTalk, Vol. 17, No. 5, pp. 4-9, 2004. Boehm, B. and Lane J., "21st Century Processes for Acquiring 21st Century Software-Intensive Systems of Systems." CrossTalk: Vol. 19, No. 5, pp.4-9, 2006. Boehm, B., and Lane, J., “Using the ICM to Integrate System Acquisition, Systems Engineering, and Software Engineering,” CrossTalk, October 2007, pp. 4-9. Boehm, B., and Lane, J., “A Process Decision Table for Integrated Systems and Software Engineering,” Proceedings, CSER 2008, April 2008. Boehm, B., “Future Challenges and Rewards for Software Engineers,” DoD Software Tech News, October 2007, pp. 6-12. Boehm, B. et al., Software Cost Estimation with COCOMO II, Prentice Hall, 2000. Boehm, B., Software Engineering Economics, Prentice Hall, 2000. Carlock, P. and Fenton, R., "System of Systems (SoS) Enterprise Systems for Information-Intensive Organizations," Systems Engineering, Vol. 4, No. 4, pp. 242-26, 2001. Carlock, P., and J. Lane, “System of Systems Enterprise Systems Engineering, the Enterprise Architecture Management Framework, and System of Systems Cost Estimation”, 21st International Forum on COCOMO and Systems/Software Cost Modeling, 2006. Checkland, P., Systems Thinking, Systems Practice, Wiley, 1980 (2nd ed., 1999). Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6, http://akss.dau.mil/dag/, 2006.http://akss.dau.mil/dag/ Department of Defense (DoD), Instruction 5000.2, Operation of the Defense Acquisition System, May 2003. Department of Defense (DoD), Systems Engineering Plan Preparation Guide, USD(AT&L), 2004. Electronic Industries Alliance (1999); EIA Standard 632: Processes for Engineering a System Galorath, D., and Evans, M., Software Sizing, Estimation, and Risk Management, Auerbach, 2006. Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976.

49. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE49©USC-CSSE References -II Highsmith, J., Adaptive Software Development, Dorset House, 2000. International Standards Organization, Information Technology Software Life Cycle Processes, ISO/IEC 12207, 1995 ISO, Systems Engineering – System Life Cycle Processes, ISO/IEC 15288, 2002. Jensen, R. “An Improved Macrolevel Software Development Resource Estimation Model,” Proceedings, ISPA 5, April 1983, pp. 88-92. Krygiel, A., Behind the Wizard’s Curtain; CCRP Publication Series, July, 1999, p. 33 Lane, J. and Boehm, B., "System of Systems Cost Estimation: Analysis of Lead System Integrator Engineering Activities", Information Resources Management Journal, Vol. 20, No. 2, pp. 23-32, 2007. Lane, J. and Valerdi, R., “Synthesizing SoS Concepts for Use in Cost Estimation”, Proceedings of IEEE Systems, Man, and Cybernetics Conference, 2005. Lientz, B., and Swanson, E.B., Software Maintenance Management, Addison Wesley, 1980. Madachy, R., Boehm, B., Lane, J., "Assessing Hybrid Incremental Processes for SISOS Development", USC CSSE Technical Report USC-CSSE-2006-623, 2006. Maier, M., “Architecting Principles for Systems-of-Systems”; Systems Engineering, Vol. 1, No. 4 (pp 267-284). Maier, M., “System and Software Architecture Reconciliation,” Systems Engineering 9 (2), 2006, pp. 146-159. Northrop, L., et al., Ultra-Large-Scale Systems: The Software Challenge of the Future, Software Engineering Institute, 2006. Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development Process: A New Look, National Academy Press, 2007. Putnam, L., “A General Empirical Solution to the Macro Software Sizing and Estimating Problem,” IEEE Trans SW Engr., July 1978, pp. 345-361. Rechtin, E. Systems Architecting, Prentice Hall, 1991. Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice,” OSD/USC Integrating Systems and Software Engineering Workshop, http://csse.usc.edu/events/2007/CIIForum/pages/program.html, October 2007. http://csse.usc.edu/events/2007/CIIForum/pages/program.html

50. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE50©USC-CSSE List of Acronyms B/LBaselined C4ISRCommand, Control, Computing, Communications, Intelligence, Surveillance, Reconnaissance CDConcept Development CDRCritical Design Review COTSCommercial Off-the-Shelf DCRDevelopment Commitment Review DIDevelopment Increment DoDDepartment of Defense ECRExploration Commitment Review EVMSEarned Value Management System FCRFoundations Commitment Review FEDFeasibility Evidence Description FMEAFailure Modes and Effects Analysis FRPFull-Rate Production GAOGovernment Accountability Office GUIGraphical User Interface

51. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE51©USC-CSSE List of Acronyms (continued) HMIHuman-Machine Interface HSIHuman-System Interface HWHardware ICMIncremental Commitment Model IOCInitial Operational Capability IRRInception Readiness Review IS&SEIntegrating Systems and Software Engineering LCOLife Cycle Objectives LRIPLow-Rate Initial Production MBASEModel-Based Architecting and Software Engineering NDINon-Developmental Item NRCNational Research Council OCOperational Capability OCROperations Commitment Review OO&DObserve, Orient and Decide OODAObserve, Orient, Decide, Act O&MOperations and Maintenance

52. University of Southern California Center for Systems and Software Engineering May 2009©USC-CSSE52©USC-CSSE List of Acronyms (continued) PDRPreliminary Design Review PMProgram Manager PRPublic Relations PRRProduct Release Review RUPRational Unified Process SoSSystem of Systems SoSESystem of Systems Engineering SSESystems and Software Engineering SWSoftware SwESoftware Engineering SysESystems Engineering Sys EngrSystems Engineer S&SESystems and Software Engineering USD (AT&L)Under Secretary of Defense for Acquisition, Technology, and Logistics VCRValidation Commitment Review V&VVerification and Validation WBSWork Breakdown Structure WMIWarfighter-Machine Interface