1. 1 Alternative Designs for a Joint C4I Capability Certification Management (JC3M) System A Student Project Gregory A. Miller Naval Postgraduate School Monterey, CA Paper 046 NAVAL POSTGRDUATE SCHOOL

2. Paper 046 2 Step #3 Integration Testing (w/o SoS rqmnts) Motivation: Acquisition system & SoS integration needs Step #1 Develop Each System in Isolation Army System X Marine System Y Air Force System A Navy System N Marine System Z Step #2 Perform Developmental Testing on Each System Joint C4I System of Systems Developers & Program Offices Testing Agencies Operating Forces Fielding Decision

3. Paper 046 3 Current SoS Testing and Fielding Step #1 Develop Systems in isolation Step #2 Perform Developmental Testing on each System Step #4 Field Joint C4I System of Systems Step #3 Perform System of Systems Testing Problems with SoS testing  No performance measurements  What architecture is appropriate? Joint C4I SoS are large and constantly changing  Testing every SoS function is impossible  Hard to determine what failure is since quality of service requirements change

4. Paper 046 4 What is the Real Problem? Plan Report Is it that DoD does not define performance measures for Joint C4I SoS? Elicit Requirements Define Thresholds Define Architecture ID Systems Live Testing Simulation Formal Report Execute

5. Paper 046 5 What’s the Solution? Plan Report Develop a System that articulates SoS capabilities, determines whether SoS as a whole supports these capabilities, and reports the results Elicit Requirements Define Performance Define Architecture ID Systems Formal Report Simulation Certification Live Testing Paper review (On paper, did each SoS component meet articulated SoS capabilities?) Execute

6. Paper 046 6 JC3M in Testing and Fielding Step #1 Develop Systems in isolation Step #2 Perform Developmental Testing on each System Step #4 Field Joint C4I System of Systems JC3M goals:  Acquire objective SoS performance measurements for acquisition and user communities  Produce decision data for stakeholders  Provide confidence in SoS performance for users Step #3 Perform System of Systems Testing Step #3 Perform JC3M (Plan, Execute, Report) (Currently unavoidable) (Replaces current SoS testing methodology)

7. Paper 046 7 Systems Engineering Process Customer Needs Problem Refinement Design Alternatives Modeling and Simulation Final Recommendation Assess Performance Analysis of Alternatives Re-Evaluate

8. Paper 046 8 Revised Problem Statement  Original problem focus: Define Threshold Values  Research revealed the true problem …  Refined problem focus: Define measures to be evaluated Problem Refinement Design Alternatives Re-Evaluate

9. Paper 046 9 Define Thresholds Revised Problem Plan Report Define performance measures for Joint C4I SoS Elicit Requirements Define Measures Live Testing Simulation Formal Report Execute Define Architecture ID Systems Paper review

10. Paper 046 10 JC3M Value Hierarchy  Developed from refined problem statement  Based on stakeholder analysis Functional Non-Functional JC3M Adaptability 4.0 Report Results 3.0 Evaluate C4I SoS Capabilities 2.0 Plan C4I SoS Evaluation 1.0 Repeatability 6.0 Usability 5.0

11. Paper 046 11 JC3M Value Hierarchy

12. Paper 046 12 Alternatives FEDOSMC3T JTEM -CTM Alternative #1 Alternative #2

13. Paper 046 13 Alternative #1 “System Capabilities Review (SCR)”

14. Paper 046 14 Alternative #2 “Functional Capability Board (FCB)”

15. Paper 046 15 Differences

16. Paper 046 16 PersonnelUseScopeMeasures FEDOSInternalPast Service testStakeholder agreement MC3TInternal + External Proof of concept Service system certification Doctrine developers & stakeholders JTEM CTM InternalModel Joint Mission Effectiveness Assessment Doctrine, System documentation SCRInternalProposed Joint capability assessment Doctrine, System documentation FCBInternal + External Proposed Joint capability assessment C4I SME panel Alternatives Summary

17. Paper 046 17 Fill in the blanks!

18. Paper 046 18 M&S Overview

19. Paper 046 19 Complete EM Percentage Traceable Measures Days to Plan Evaluation Planning Output Quality Labor Elasticity Duration Elasticity %DaysLikert Scale 1-4 Unitless Ideal Value100%Less is better4 is IdealLess is better FEDOS 01403.170.87 MC3T 721213.250.78 JTEM CTM 92733.421.040.83 SCR 921583.000.98 FCB 881272.750.72

20. Paper 046 20 Value Modeling Overview Translation of raw measurements into a normalized set of weighted values that can be added.

21. Paper 046 21 LCCE – Cost Summary Alternatives Life-Cycle Year Total Cost ($) 1234…910 FEDOS 1,052,527419,497 52,2005,010,706 MC3T 1,169,414525,537 52,2005,975,913 JTEM-CTM 1,030,0002,470,0001,169,414558,53552,2006,972,824 FCB 2,323,117650,223 52,2008,127,101 SCR 2,121,421624,451 52,2007,719,232 Interpretation: The delta between the highest and lowest LCCE ≈ $3M, which is not a significant sum over a ten year span.

22. Paper 046 22 Utility & LCCE Percentage of Traceable Measures Days to Plan Evaluation Quality of Planning Outputs Elasticity of Labor Elasticity of Duration Overall Utility (0 – 1) LCCE ($ M) FEDOS 0.000.040.390.060.140.635.01 MC3T 0.020.050.390.070.170.715.98 JTEM CTM 0.240.060.400.040.150.896.97 SCR 0.240.020.370.050.100.797.72 FCB 0.220.050.340.080.180.878.13

23. Paper 046 23 LCCE vs Utility

24. Paper 046 24 Way Ahead: 3 areas Capability Portfolio Manager Functional Capability Board Likelihood Impact