1. © 2011 Carnegie Mellon University Should-Cost: A Use for Parametric Estimates Additional uses for estimation tools Presenters:Bob Ferguson (SEMA) Date:November 3, 2011 Location:COCOMO Forum 2011

2. 2 © 2011 Carnegie Mellon University “Should-Cost” Background Ashton Carter Memo, September 14, 2010: Driving for affordability by enrolling contractors in mid-stream process improvement.

3. 3 © 2011 Carnegie Mellon University Methods Used to Analyze Contractor Processes Activity based costing (Consulting firm) Audit or use contractor’s data to allocate costs to activity Roll-up activity by Role (programmer, analyst, …), Process (design, …) and Technology (language, database, platform …) Compare to benchmarks Parametric Estimation (what I did) Use data from contractor’s basis of estimate (size, productivity, lifecycle) Construct parametric estimate that (nearly) duplicates contractor’s estimate. Test sensitivity of parameters and estimate benefits of improvement Process Assessment (not used) Based on detailed interviews and artifact analylsis

4. 4 © 2011 Carnegie Mellon University Methods and Data for Parametric Review Methods Park, R., A Manager's Checklist for Validating Software Cost and Schedule Estimates, SEI-95-SR-004 SEER-SEM, COCOMO parametric estimation tools used Park’s checklist Checklist used to see whether I had sufficient information to reproduce the estimate. Answers the question: Can I trust the estimate? Process, estimating experience, documented assumptions, … Data used (COCOMO and SEER are similar): Software size (LOC) by subsystem with new, modified, reused and base Product complexity and unprecedentness factors (embedded vs. simulation…) People and team capabilities and experience Benchmarked performance (CMMI, or other)

5. 5 © 2011 Carnegie Mellon University Results of Analyzing 11 Estimates 1 – Estimated schedule and cost corresponded to very high productivity and quality so there was little room for cost improvement (~5%). 2 – Very good in productivity (<10%) 4 – Estimate was padded by assuming people would not be as productive compared to previous products. (~15%) 4 – Very difficult to determine a basis of estimate because insufficient detail about size was presented. Comparison to industry for productivity and quality

6. 6 © 2011 Carnegie Mellon University Why Both SEER & COCOMO SEER Component level breakdown could be performed Sensitivity analysis reports One of the contractors was using SEER so BOE was visible COCOMO One contractor did not provide component level detail. It was difficult to understand their usage of re-use vs. modified vs. base code. Simplified assumptions were easier to apply to COCOMO. SEER takes longer to edit for iteration of multiple estimates. My own inexperience with SEER.

7. 7 © 2011 Carnegie Mellon University Should-Cost Results Improved value Up to 47% improvement on projected component cost – overall 22%! At least modest cost improvement (~5%) identified on every component evaluated (all 11 estimates). Some improvement suggestions (first 2 related to parameters)  Most contractors claimed low cohesion on teams. People pulled to fight fires.  Some contractors wanted to claim “inexperience” in domain and claim “best in class” salaries in the same breath. Systems integration needs better synchronization. Added overhead appeared directly in the BOE documentation. Delays waste valuable resources and result in higher costs.. Project management overhead was too high for every contractor. Program office needs both better processes and more technical personnel. Product quality problems existed in prior efforts.

8. 8 © 2011 Carnegie Mellon University Conclusions Parametric estimates are a useful method for should-cost evaluations. It was important to validate the BOE first as otherwise contractor can challenge the conclusions. Parametric method should be supplemented with bottom-up evaluation. Non-software services and products must be included. Product and Process Quality must be checked (e.g. estimation of defects) Interfaces between contractor and program office are a critical source of data. Benchmark data sources are required. Estimation tools Capers Jones’ books …

9. 9 © 2011 Carnegie Mellon University Contact Information Presenters/ Points of Contact Dave Zubrow SEMA Telephone: +1 412-268-5243 Email: dz@sei.cmu.edudz@sei.cmu.edu Bob Ferguson SEMA Telephone: +1 412-268-9750 Email: rwf@sei.cmu.edurwf@sei.cmu.edu Web: www.sei.cmu.edu www.sei.cmu.edu/measurement/ U.S. mail: Software Engineering Institute Customer Relations 4500 Fifth Avenue Pittsburgh, PA 15213-2612 USA Customer Relations Email: info@sei.cmu.edu Telephone: +1 412-268-5800 SEI Phone: +1 412-268-5800 SEI Fax: +1 412-268-6257

10. 10 © 2011 Carnegie Mellon University NO WARRANTY THIS CARNEGIE MELLON UNIVERSITY AND SOFTWARE ENGINEERING INSTITUTE MATERIAL IS FURNISHED ON AN “AS-IS" BASIS. CARNEGIE MELLON UNIVERSITY MAKES NO WARRANTIES OF ANY KIND, EITHER EXPRESSED OR IMPLIED, AS TO ANY MATTER INCLUDING, BUT NOT LIMITED TO, WARRANTY OF FITNESS FOR PURPOSE OR MERCHANTABILITY, EXCLUSIVITY, OR RESULTS OBTAINED FROM USE OF THE MATERIAL. CARNEGIE MELLON UNIVERSITY DOES NOT MAKE ANY WARRANTY OF ANY KIND WITH RESPECT TO FREEDOM FROM PATENT, TRADEMARK, OR COPYRIGHT INFRINGEMENT. Use of any trademarks in this presentation is not intended in any way to infringe on the rights of the trademark holder. This Presentation may be reproduced in its entirety, without modification, and freely distributed in written or electronic form without requesting formal permission. Permission is required for any other use. Requests for permission should be directed to the Software Engineering Institute at permission@sei.cmu.edu.permission@sei.cmu.edu This work was created in the performance of Federal Government Contract Number FA8721-05-C-0003 with Carnegie Mellon University for the operation of the Software Engineering Institute, a federally funded research and development center. The Government of the United States has a royalty-free government-purpose license to use, duplicate, or disclose the work, in whole or in part and in any manner, and to have or permit others to do so, for government purposes pursuant to the copyright license under the clause at 252.227-7013.