1. Review of Effectiveness Measures Evaluation Task Dan Ingold, USC USC-CSSE Annual Research Review March 16, 2009

2. EM Task Statement of Work Develop measures to monitor and predict system engineering effectiveness for DoD Major Defense Acquisition Programs – Define SysE effectiveness – Develop measurement methods for contractors, DoD program managers and PEOs, oversight organizations For weapons platforms, SoSs, Net-centric services – Recommend continuous process improvement approach – Identify DoD SysE outreach strategy Consider full range of data sources – Journals, tech reports, org’s (INCOSE, NDIA), DoD studies Partial examples cited: GAO, SEI, INCOSE, Stevens/IBM GFI: Excel version of SADB Deliverables: Report and presentation – Approach, sources, measures, examples, results, recommendations March 16, 20092

3. Target EM Task Benefits for DoD Identification of best available EM’s for DoD use – Across 3 domains; 3 review levels; planning and execution Early warning vs. late discovery of SysE effectiveness problems Identification of current EM capability gaps – Recommendations for most cost-effective enhancements, research on new EM approaches – Ways to combine EM strengths, avoid weaknesses Foundation for continuous improvement of DoD SysE effectiveness measurement – Knowledge base of evolving EM cost-effectiveness – Improved data for evaluating SysE ROI 01/29/20093

4. Measuring SysE Effectiveness- And measuring SysE effectiveness measures Good SysE correlates with project success – INCOSE definition of systems engineering, “An interdisciplinary approach and means to enable the realization of successful systems” Good SysE not a perfect predictor of project success – Project does bad SysE, but gets lucky at the last minute and finds a new COTS solution, producing a great success – Project does great SysE, but poor managers and developers turn it into a disaster Goodness of a candidate SysE effectiveness measure (EM) – Whether it can detect when a project’s SysE is leading the project more toward success than toward failure Heuristic for evaluating a proposed SysE EM – Role-play as underbudgeted, short-tenure project manager – Ask “How little can I do and still get a positive rating on this EM?” March 16, 20094

5. Candidate Measurement Methods NRC Pre-Milestone A & Early-Phase SysE top-20 checklist Army Probability of Program Success (PoPS) Framework INCOSE/LMCO/MIT Leading Indicators Stevens Leading Indicators (new; using SADB root causes) USC Anchor Point Feasibility Evidence progress UAH teaming theories NDIA/SEI capability/challenge criteria SISAIG Early Warning Indicators/ USC Macro Risk Tool March 16, 20095

6. Independent EM Evaluations and Resolution March 16, 20096 Candidate EM USC Stevens FC-MD UAH PoPS Leading Indicators X X X INCOSE LIs X X Stevens LIs X X X SISAIG LIs/ Macro Risk X X X NRC Top-20 List X X X SEI CMMI-Based LIs X X X USC AP-Feasibility Evidence X X X UAH Team Effectiveness X X X

7. Candidate EM Coverage Matrix March 16, 20097 SERC EM Task Coverage Matrix V1.0 NRC Probability of Success SE Leading Indicators LIPSF (Stevens) Anchoring SW Process (USC) PSSES (U. of Alabama) SSEE (CMU/SEI) Macro Risk Model/Tool Concept Dev Atleast 2 alternatives have been evaluated xxx x (w.r.t NPR) (x) Can an initial capability be achieved within the time that the key program leaders are expected to remain engaged in their current jobs (normally less than 5 years or so after Milestone B)? If this is not possible for a complex major development program, can critical subsystems, or at least a key subset of them, be demonstrated within that time frame? x(x)x x (5 years is not explicitly stated) (x) (seems to be inferrable from the conclusions) (x) (implies this) Will risky new technology mature before B? Is there a risk mitigation plan? xxx(x)xx Have external interface complexities been identified and minimized? Is there a plan to mitigate their risks? xxxxxx KPP and CONOPS At Milestone A, have the KPPs been identified in clear, comprehensive, concise terms that are understandable to the users of the system? x(x)x x (strongly implied) (x) (implied) xx At Milestone B, are the major system-level requirements (including all KPPs) defined sufficiently to provide a stable basis for the development through IOC? xx(x)xx (x) (There is no direct reference to this but is inferrable) x Has a CONOPS been developed showing that the system can be operated to handle the expected throughput and meet response time requirements? xx(x) x (x) (there is a mention of a physical solution. That's the closest in this regard) xx Legend: x = covered by EM (x) = partially covered (unless stated otherwise)

8. EM Evaluation First cut is to complete “45 x 8” evaluation Evaluation identifies key criteria / EMs Preliminary coverage & commonality eval – Four EMs cover more than 1/2 the criteria – Top-20 criteria mentioned by at least 5 EMs Revise results after team evaluations done March 16, 20098

9. New SE Evaluation Forms March 16, 20099

10. EM Coverage March 16, 200910 Effectiveness Measure Number Covered Percent Covered SEI SSEE36 80% NRC30 67% USC Anchor Point28 62% SISAIG/Macro Risk27 60% UAH PSSES21 47% Army PoPS19 42% Stevens LI-PSF18 40% INCOSE SE LI17 38%

11. EM Commonality March 16, 200911 # of Mentions Effectiveness Measures 85, 6, 7, 15 78, 9, 11, 18, 19 63, 4, 24, 25, 39, 42 51, 2, 10, 17, 29 412, 13, 16, 23, 43, 45

12. Most Mentioned (8) March 16, 200912 EMDefinition 5KPPs identified in concise, understandable terms 6KPPs sufficiently stable for development 7CONOPS shows system handles throughput and response time 15Key risk drivers (not just technical) are identified

13. Most Mentioned (7) March 16, 200913 EMDefinition 8Major cost / schedule drivers identified, risk plan in place 9Cost confidence level accepted by stakeholders 11Requirements account for likely future mission growth 18Top-level plan defined for system integration and testing 19Sufficient experienced and talented program and SE managers identified

14. Most Mentioned (6) March 16, 200914 EMDefinition 3Risky new technology will mature before MS B, and risk mitigation plan in place 4External interface complexities identified and minimized, risk mitigation plan in place 24Process compliance 25Review of review process 39Verification / validation and configuration management 42Program governance process, SE plan well articulated

15. Most Mentioned (5) March 16, 200915 EMDefinition 1At least two alternatives have been evaluated 2Initial capability can be achieved within expected duty rotation (< 5 years) 10Sufficient collection of models to validate CONOPS against KPPs 17Plan in place at MS A defining MS B activities and performers 29Level of service has been validated

16. Test of EM Evaluation Process March 16, 200916

17. New Evaluation Framework Synthesized from workshop crosswalks Simpler than SEPP / SISAIG / 45 x 8 – Four major categories – Four to five elements per category Provides taxonomy of existing frameworks Coverage spans previous frameworks Could be basis for new Macro Risk-like tool March 16, 200917

18. Evaluation Framework Criteria March 16, 200918

19. SE Competencies Developed by ODNI Comprehensive survey of core competencies – 10 candidate work activities – 173 candidate knowledge, skills & abilities (KSAs) To our knowledge, not yet validated Approved for limited release within SERC March 16, 200919

20. SE Competency Sample March 16, 200920

21. Q & A March 16, 200921