1. © The Aerospace Corporation 2010 Life Cycle Alignment Concerns in Acquisition of Software-Intensive Systems Dr. Peter Hantos and Nancy Kern The Aerospace Corporation 1 2010 COCOMO International Forum, Los Angeles, California November 1-4, 2010

2. 2 Acknowledgements This work would not have been possible without the following: –Reviewers Suellen Eslinger Zane Faught Rick Johnson Dr. Leslie Holloway –Sponsors Rosalind Lewis Dr. Malina Hills –Funding Source The Aerospace Corporation’s Independent Research & Development Program

3. 3 Outline Objectives Introduction Basic Lifecycle Alignment Concerns Alignment Concerns for Incremental Development and Evolutionary Acquisition … and Where Everything is Really Falling Apart Conclusions Acronyms References Referenced Standards and Government Resources

4. 4 Objectives The presentation’s objective is to lay the foundations for the effective tailoring and use of Life Cycle Process Standards in Software-Intensive System Acquisition The focus of inquiry is on two interrelated problems: –The prevailing technical standards, i.e., systems engineering life cycle processes and software life cycle processes, that should be guiding the contractors’ development efforts are not properly aligned –These technical standards and the DOD 5000.02 Defense Acquisition Framework are not properly aligned either Disclaimer –Discussion of hardware concerns and detailed tailoring guidance are out of scope for this presentation

5. 5 Introduction

6. 6 Basic Process Relationships Legend: HW – Hardware SW – Software RFP – Request for Proposal Clear delineation of acquisition and engineering processes

7. 7 Basic Expectation Regarding Life Cycle Processes and Life Cycle Models Life cycle process standards are harmonized, life cycle models are aligned

8. 8 Why the Emphasis on Life Cycle Process Standards? Our primary focus is Mission Success [MAG 2007] -Mission Success is defined as the achievement by an acquired system (or system of systems) to singularly or in combination meet not only specified performance requirements but also the expectations of the users and operators in terms of safety, operability, suitability and supportability -Mission Assurance is the disciplined application of general systems engineering, quality, and management principles towards the goal of achieving Mission Success, and, towards this goal, this disciplined application provides confidence in its achievement How can it be ensured that high mission assurance processes are used to develop the objective system? –Use a robust development standard [Eslinger 2006] Note that even the use of so-called mature processes that are based on such frameworks as the CMMI ® is inadequate, and the government must require contractual compliance with a robust development standard The foundation for this conclusion has been derived from the analysis of Acquisition Reform-induced failures on numerous space programs ® CMMI is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University

9. 9 Basic Alignment Concerns

10. 10 The Defense Acquisition Life Cycle Model DOD 5000.02 represents a System Development Life Cycle tailored for the Defense Acquisition context; life cycle phases are separated by acquisition milestones (acquisition decision points) –However, it also mandates systems engineering phase-gates (reviews) –Red, curved arrows indicate how technical reviews support acquisition milestones The model prescribes processes for both the Acquisition Program Offices (APOs) and the contractors Unfortunately, its structure and language are hardware-oriented –Software interpretation is frequently ambiguous

11. 11 Aligning ISO/IEC 15288-2008 with DOD 5000.02 Systems Engineering starting point is clear: Milestone A However, beginning of Development is ambiguous (More on this later) End of Development is clear: OTRR ISO/IEC 15288-2008 does not account for additional transitional effort if the system will be operated by another entity * Source: [Valerdi 2008] *

12. 12 Critical, Mandated Contractor Activity Not Accounted for in ISO/IEC 15288-2008 The DOD is very concerned about the technologies associated with the acquired systems and conducts technology assessments beginning at Milestone A –Prior to SFR, all competing contractors must carry out aggressive technology risk reduction activities and technology maturity demonstrations via prototyping [Dahmann 2009] These activities require substantial contractor effort; still, likelihood of success is highly uncertain

13. 13 Aligning ISO/IEC 12207-2008 with ISO/IEC 15288-2008 for Software-Intensive Systems SW support activities required for system conceptualization are missing System requirements supposed to be allocated and a high-level system architecture should be available before SW requirements allocation SW development should end prior to System Integration & Test

14. 14 Using ISO/IEC 12207-2008 for Software-Only Systems The beginning of SW Development is clearly aligned with MS A Unless the process is Waterfall, there is no designated System Integration and Test period at the end of Software Development - Software is integrated continuously, on a build by build basis - However, software must be completely integrated and tested by OTRR Unfortunately, ISO/IEC 12207 and ISO/IEC 15288 are still inadequately harmonized Examples on the next slide

15. 15 Selected Alignment Concerns for ISO/IEC 12207:2008(E) and ISO/IEC 15288-2008* Implementation –Clause 6.4.4 Implementation Process 6.4.4.1 The purpose of the Implementation Process is to realize a specified system element –Clause 7.1.1 Software Implementation Process “NOTE The Software Implementation Process is a conforming instance of the Implementation Process of ISO/IEC 15288…” –Concern: Implementation in ISO/IEC 15288 should refer only to systems engineering activities, not activities of all contributing (i.e., software) disciplines The back-end of the Software Development Process –Clause 6.4.8 Software Acceptance Support Process “NOTE The Software Acceptance Support Process … contributes to the outcomes of the Transition Process of ISO/IEC 15288. Users may consider claiming conformance to 15288 rather than the process in this standard.” –Concern: This is a recurring theme, i.e., claiming conformance selectively and alternatively to the different standards * Referenced quotes from ISO/IEC 12207:2008(E) The relationship between the two standards is still ambiguous

16. 16 Alignment Concerns for Incremental Development and Evolutionary Acquisition

17. 17 Incremental System Development Actual planning of incremental system development can start after SFR –Planning of increments is a substantial, additional activity –All players must plan but only the winning contractor will actually continue Every increment goes through all of the mandated technical reviews By the time the program is to go to the Operation & Support acquisition phase, the last increment must reach IOC Increment Planning

18. 18 Incremental Software Development Note that the chart shows Waterfall methodology for increment development; however, it can be any approach (e.g., iterative) Increments are successively integrated with each other Regression testing of software increments is necessary The final increment is migrated to the System Integration & Test environment No standard methodology for increment-level reviews

19. 19 Evolutionary Acquisition (High Technology Readiness) The contractor’s technical effort to support the government’s upgrade decision is not reflected in ISO/IEC 15288-2008 –Note that this effort is not the same as what “Enhance” refers to Contractor support to upgrade decision Third Acquisition Increment is Initiated

20. 20 Evolutionary Acquisition (Low Technology Readiness) Again, the contractor’s technical effort to support the government’s upgrade decision is not reflected in ISO/IEC 15288-2008 As was shown, all Critical Technology Elements (CTEs) must reach the mandated Technology Readiness Levels (TRLs) by Milestone B –This maturation effort is not reflected in ISO/IEC 15288-2008 Contractor support to upgrade decision

21. 21 … and Where Everything is Really Falling Apart

22. 22 Systems Engineering Process This is supposedly the complete Systems Engineering Process (SEP)* –However, some sources only view it as the Requirements Development Process [EIA/IS 632] Naming games with scope and content-related consequences –Systems Engineering Process [INCOSE 2003, MIL-STD-499B] vs. Processes for Engineering a System (EIA/IS 632) Due to the iteration loops it is very difficult to establish stage gates The “Synthesis” part is ill-defined –The applicable Life Cycle Process Standards, ISO/IEC 15288, ISO/IEC TR 19760, or ISO/IEC 12207, do not even mention it * Source: [INCOSE 2003], [MIL-STD-499B]. Recursive & Iterative… Recursive & Iterative…

23. 23 Contractor Technical Reviews In numerous, previous slides it was shown that the positioning of the system-level reviews is ambiguous –Originally they were meant to be phase-gates of a sequential process. However, while the acquisition life cycle is indeed sequential, the underlying technical processes are not. Consequently, the positioning of all technical reviews is arbitrary The content of the system-level reviews is ambiguous –The expected content of the reviews is different in different organizations –All references to reviews, e.g., the descriptions in the Defense Acquisition Guidebook, are modeled after MIL-STD-1521B; However, that standard is now obsolete –MIL-STD-1521B assumed a sequential development life cycle; Since the development processes are not sequential anymore, review content is ill- defined and ambiguous There is no standard methodology for increment-level (build) reviews Unfortunately, the situation has not really change since 2004 –For a detailed analysis of reviews, see [Hantos 2004]

24. 24 Conclusions Life cycle models and life cycle processes have a fundamental role in program management Mission assurance considerations also warrant the use of robust life cycle process standards on acquisition contracts However, the use of inadequate or out-of-synch life cycle processes represents a major program management risk To mitigate this risk, a thorough understanding and careful tailoring of the involved technical standards are needed

25. 25 Acronyms

26. 26 References Dahmann 2009 Dahmann, J.S., and Kelley, M., Systems Engineering During the Materiel Solution and Technology Development Phases, Washington, DC: Office of the DDR&E/Systems Engineering, 2009, www.acq.osd.mil/sse Eslinger 2006 Eslinger, S., Mission Assurance-Driven Processes for Software-Intensive Ground Systems, The Aerospace Corporation Technical Report ATR-2006(8056)-1, September 30, 2006 Hantos 2004 Hantos, P., System and Software Reviews Since Acquisition Reform, Southern California Software Process Improvement Network, April 2, 2004 INCOSE 2003 INCOSE Systems Engineering Handbook, INCOSE-TP-2003- 016-02, Version 2a, June 1, 2004 MAG 2007 Mission Assurance Guide, The Aerospace Corporation Technical Operating Report, TOR-2007(8546)-6018 Rev. A, July 1, 2007 Valerdi 2008 Valerdi, R., The Constructive Systems Engineering Cost Model (COSYSMO), VDM Verlag Dr. Mueller, 2008

27. 27 Referenced Standards and Government Resources Defense Acquisition Guidebook, Fort Belvoir, VA, Defense Acquisition University, DOD 5000.02, Operation of the Defense Acquisition System, December 8, 2008 EIA/IS 632-1988, Processes for Engineering a System ISO/IEC 12207:2008(E), Systems and Software Engineering - Software life cycle processes ISO/IEC 15288-2008(E), Systems Engineering – System life cycle processes ISO/IEC TR 19760:2003(E), Systems Engineering – A guide for the application of ISO/IEC 15288 (System life cycle processes) MIL-STD-499B, Draft Military Standard Systems Engineering, September 7, 1993 MIL-STD-1521B, Military Standard Technical Reviews and Audits for Systems, Equipments, and Computer Software, June 4, 1985

28. 28 Use of Trademarks, Service Marks and Trade Names Use of any trademarks in this material is not intended in any way to infringe on the rights of the trademark holder. All trademarks, service marks, and trade names are the property of their respective owners. Author of clip art on Slide 22 is Elee Kirk. Used with permission.