1. Using Parametric Software Estimates During Program Support Reviews Chris Miller Office of the Deputy Director, Software Engineering and System Assurance S YSTEMS & S OFTWARE E NGINEERING Office of the Deputy Under Secretary of Defense for Acquisition and Technology US Department of Defense October 2008 Version 2.0

2. Systems & Software Engineering – Chris Miller, October 2008 Slide 2 OUSD (AT&L) Organization May 2006 USD, Acquisition Technology & Logistics DUSD, Acquisition & Technology Dir, Joint Advanced Concepts Dir, Systems and Software Engineering Dir, Portfolio Systems Acquisition Industrial Programs Defense Procurement and Acquisition Policy Small Business Programs Defense Contract Management Agency Defense Acquisition University

3. Systems & Software Engineering – Chris Miller, October 2008 Slide 3 Director, Systems & Software Engineering Systems and Software Engineering Director, Systems & Software Engineering Deputy Director Enterprise Development Deputy Director Developmental Test & Evaluation Deputy Director Software Engineering & System Assurance Deputy Director Assessments & Support

4. Systems & Software Engineering – Chris Miller, October 2008 Slide 4 Elements of a DoD Strategy for Software Support Acquisition Success –Ensure effective and efficient software solutions across the acquisition spectrum of systems, SoS and capability portfolios Improve the State-of-the-Practice of Software Engineering –Advocate and lead software initiatives to improve the state-of-the- practices through transition of tools, techniques, etc. Leadership, Outreach and Advocacy –Implement at Department and National levels, a strategic plan for meeting Defense software requirements Foster Software Resources to meet DoD needs –Enable the US and global capability to meet Department software needs, in an assured and responsive manner Promote World-Class Leadership for Defense Software Engineering

5. Systems & Software Engineering – Chris Miller, October 2008 Notional View of Software Measurement Software Engineering and Systems Assurance (SSA) initiatives Program feasibility analysis using estimation models Software Resources and Data Report: Feasibility Study Revision of MIL-HDBK-881A to improve software guidance Integration of software metrics with EVM to assess consistency of estimates Concepts - Requirements - Arch/Design Development - Maintenance Determine methods of obtaining cost estimating data Use estimation tools, techniques, processes, & practices Generate SW appropriate WBS Generate SW appropriate WBS Earned Value Management (EVM) for SW Earned Value Management (EVM) for SW Link SW quality indicators to EVM

6. Systems & Software Engineering – Chris Miller, October 2008 6 Driving Technical Rigor Back Into Programs “Program Support Reviews” Program Support Reviews provide insight into a program’s technical execution focusing on: –SE as envisioned in program’s technical planning –T&E as captured in verification and validation strategy –Risk management - integrated, effective and resourced –Milestone exit criteria as captured in Acquisition Decision Memo –Acquisition strategy as captured in Acquisition Strategy Report Independent, cross-functional view aimed at providing risk-reduction recommendations The PSR reduces risk in the technical and programmatic execution of a program

7. Systems & Software Engineering – Chris Miller, October 2008 7 Program Support Review Activity (as of August 2008) PSRs/NARs completed: 57 AOTRs completed: 13 Nunn-McCurdy Certification: 13 Participation on Service-led IRTs: 3 Technical Assessments: 13

8. Systems & Software Engineering – Chris Miller, October 2008 8 DoD Software Performance: What We’re Seeing* Software issues are significant contributors to poor program execution –Schedule realism (compressed, overlapping) –Software requirements not well defined, traceable, testable –Immature architectures, COTS integration, interoperability, obsolescence (electronics/hardware refresh) –Software development processes not institutionalized, planning documents missing or incomplete, reuse strategies inconsistent –Software test/evaluation lacking rigor and breadth –Lessons learned not incorporated into successive builds –Software risks/metrics not well defined, managed *Based on over 60 program reviews over the past 3 ½ years

9. Systems & Software Engineering – Chris Miller, October 2008 9 Software Engineering and System Assurance (SSA) Role SSA produced software estimates to support several PSR teams in 2007 –Program A: SSA was asked to assess software schedule feasibility prior to MS B –Program B: Significant software issues Opportunity for SSA to support program decision making by providing software estimates –Estimation activities aimed at gauging overall program feasibility and quantifying magnitude of top program risks –Focus on support for engineering vs. budgeting decisions

10. Systems & Software Engineering – Chris Miller, October 2008 10 Program A PSR of aircraft program with ambitious schedule –Three years (36 months) from Milestone B to LRIP –Modifications needed to meet U.S. requirements Developed three software estimates –Software size (SLOC) estimates provided by program office –Re-estimated both new and reused code »Based on reuse and code growth »Estimates for optimistic, typical, and pessimistic were identified as “Min”, “Mid”, and “Max” –Used three parametric models »COCOMO II, SLIM, and SEER-SEM –Most input parameters set at nominal

11. Systems & Software Engineering – Chris Miller, October 2008 11 Program A Estimation & Feasibility Analysis –Given an adjusted code size (ASIZE) of 918K to 1590K SLOCs –A range of 5375 to 9571 person months should be expected over a 62 to 74 calendar month schedule –All three models forecasted 65 to 68 months (assuming a 50% confidence-level) Result: –Analysis revealed existing acquisition strategy was not feasible –Service added more schedule to acquisition strategy –DUSD(A&T) estimates change in acquisition strategy saved $5 billion

12. Systems & Software Engineering – Chris Miller, October 2008 12 Program B Major subcontractor with significant software content did not use any parametric estimating tool –Firm Fixed Price subcontract At one point there was a substantial conflict on estimates between major subcontractor and prime, related to requirements –One portion of software grew from ~250KSLOC to 863KSLOC (3.5x), reflecting “shall not degrade current capability” requirement Increase in software engineering staff by 57 percent (40 people) over 9 month period –Based on schedule, contractor should be drawing down software staff Significant variation in code estimates during review

13. Systems & Software Engineering – Chris Miller, October 2008 13 CSCISupplierOriginalNM Review2 wks. laterNewReuseComplete %Test coverage to-date DR's/Problem reports 1A40,000374,739250,844201,14449,70075%68%47 2B27,00050,363 40,36310,00090%85%0 3A20,00038,962 071%60%5 4 C325,000415,249401,732131,921269,81198% 1 55 60 7D65,000283,25525,46718,3677,10082%60%0 8C33,000373,587100,12968,12932,00099% 0 9E15,0007,879 2,0005,879100% 1 10F114,000218,609 16,409202,20099%95%0 11G26,00025,62234,5448,92225,622100% 2 12H6,00016,580 15,5801,000100% 2 13H2,00034,806 17,20617,600100% 0 14I33,00042,355 31,65510,70099%97%1 15G15,000126,238119,6266,433113,193100% 0 16K23,00026,404 2,60423,80091% 1 17L7,00012,500 1,00011,500100% 1 18C1,00029,121 25,6213,500100% 7 19MNA1001,000 Unavail100% 0 20CNA2,600 4002,20081%43%0 752,0002,078,9691,413,521 Program B Changes in SLOC estimates…

14. Systems & Software Engineering – Chris Miller, October 2008 14 Program B Schedule Estimates Time from Detailed Requirements Complete Through End of Developmental Testing 75 KSLOC remaining based on last estimate

15. Systems & Software Engineering – Chris Miller, October 2008 15 Program B Estimation and Feasibility Analysis –Contractor appears to be driven to meet schedules versus costs »Significant increase in effort for minimal schedule savings –Uncertainty in scope of remaining software development »Need to develop firmer size estimates Recommendations –PMO reach a decision on unstable requirements to prevent further code growth – in or out –Program office bring in parametric estimating consultant to review contractor’s estimates for most volatile software components Result –Acquisition Decision Memorandum requires Service to conduct software review

16. Systems & Software Engineering – Chris Miller, October 2008 16 Integrating Management Indicators Estimation & Planning Measurement & Analysis Risk Management SDP: WBS, IMP, IMS, Metric Definitions Data & Insight Prioritized Risks Risk Mitigation Status Program Context Insight Needed

17. Systems & Software Engineering – Chris Miller, October 2008 SSA Software Measurement & Analysis Initiatives Software EVM Study/Pilot –Focus: Development of cost controls for software component of program –Execution: Partnering with DCMA, ARA, PA&E, Contractors Cost Estimation / SRDR Feasibility Study –Focus: Analysis on software cost estimation practices, availability of historical data, and increasing confidence in estimates –Execution: Partnering with PA&E/DCARC and Hill AFB STSC to evaluate existing data sources Work Breakdown Structure Handbook Update –Focus: Recommendations for update of MIL HDBK 881 to improve handling of software –Execution: Partnering with Services, DCMA, ARA, DAU, NDU, PA&E/DCARC and using NDIA expert panel

18. Systems & Software Engineering – Chris Miller, October 2008 Software EVM Study/Pilot Development of a methodology combining EVM and software metrics analysis to predict cost and schedule overruns Piloting on a 5-year ACAT 1D SW development program, as part of a cross-cutting study Pilot indicator shows ETC (estimate-to-complete) forecasts for: –Existing program management plans –EVM measures –Software metrics (i.e, growth profile of size, effort, and defects) Equivalent ETC forecasts provides an increased confidence in project plans

19. Systems & Software Engineering – Chris Miller, October 2008 Integrated EVM Concept SW Metric ETC Program Plan ETC EV ETC Acquisition Oversight confidence increases as ETCs overlap i.e., as multiple measures indicate to same conclusion

20. Systems & Software Engineering – Chris Miller, October 2008 SRDR Feasibility Study SSA funded a STSC feasibility study on normalizing DCARC data –Purpose of the study was to determine if the DCARC data set could be used to develop a parametric software estimation model –Dr. Randall Jensen at Hill AFB is the creator of Sage and SEER- SEM Preliminary study results –Data was submitted using a wide variety of definitions that made it difficult to use directly, but was successful in normalizing the data and validating the results –The data collected in the SRDR forms allows a first-level parametric model to be constructed (lack of environment information prohibits the creation of a detailed model) –Minimum development time projection model

21. Systems & Software Engineering – Chris Miller, October 2008 Minimum Software Development Time Projection Model Source: Long, L. G. et al, Aerospace Corp Report,2004 Impossible Zone Analysis of CSCI data: Apparent Maximum Size of 200 KSLOC (per CSCI) Apparent Minimum Schedule (Impossible Zone) Environment & Complexity are primary drivers of variation Apparent Maximum Schedule of 50 months

22. Systems & Software Engineering – Chris Miller, October 2008 Updating MIL-HDBK-881A for Software Overview Revision objectives: –Revise to make handbook acceptable of software engineering practice –Correct errors Notable changes: –Replace 'material item' with ' acquisition program' –Add words to make 'artifacts' equal to 'products' –Include words that make 'product-oriented' and 'DoDAF architecture' views interchangeable WRT to creating a WBS Results –Compiled a Comment matrix –Drafted a new Appendix B (for software) During final review and clean up, the working group was made aware of 90’s effort (draft MIL-HDBK-171)

23. Systems & Software Engineering – Chris Miller, October 2008 MIL-HDBK-881A Next Steps The Software Cost Control working group goal is to provide an community-coordinated set of recommendations Reached out to industry members of NDIA to review the suggested changes –Ensure recommended changes are improvements from industry perspective –Obtain additional examples to include in Appendix B –NDIA feedback is due on September 19th Submission of suggested changes to PA&E ARA will take place after collection of NDIA feedback and consolidate into a single baseline

24. Systems & Software Engineering – Chris Miller, October 2008 24 Questions/Discussion Contact Information: Chris Miller SSE/SSA Support Software Engineering & Systems Assurance ODUSD(A&T) Systems & Software Engineering Christopher.miller.ctr@osd.mil