1. 1 C2 Maturity Model Experimental Validation Statistical Analyses of ELICIT Experimentation Data Dr. David S. Alberts

2. 2 Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

3. 3 Background SAS-065 has developed a C2 Maturity Model for NATO Network Enabled Capability SAS-065 is engaged in a variety of activities to “validate” this model –Case studies: Analysis of historical complex endeavors to establish construct validity –Analysis of Experimentation Data to test a set of maturity model hypotheses ELICIT is an experimentation platform that instruments the actions of a group of participants engaged in a situational awareness problem ELICIT experiments have explored differences between “edge” and “hierarchical” organizations

4. 4 ELICIT Scenario The goal of each set of participants is to build situational awareness and identify the who, what, when, and where of a pending attack –Participants can share factoids directly with each other or post factoids to websites –Participants build awareness by gathering and analyzing factoids and interacting with one another –No one is given sufficient information to solve their assigned problem without receiving information from others The receiving, sharing, and posting of factoids and the nature of the interactions between and among participants can be constrained C2 approach for this series of experiments were designated prior to the start of the run as Hierarchy or Edge Hierarchy Edge

5. 5 Hypotheses “Hierarchical” organizations as instantiated in ELICIT experiments correspond to De-Conflicted C2 in the NATO C2 Maturity Model “Edge” organizations as instantiated in ELICIT experiments correspond to a higher level of maturity in the NATO C2 Maturity Model Hypotheses: –Edge organizations exhibit the behaviors associated with Coordinated and Collaborative levels of C2 Maturity –Hierarchical organizations exhibit the behaviors associated with the De-conflicted level of C2 Maturity –Edge organizations exhibit higher levels of effectiveness and efficiency than Hierarchical organizations

6. 6 ELICIT Data Set Includes data from 37 ELICIT experimentation trials Venues –Boston Univ. (2 runs) –NPS (16 runs) –Portugal (6 runs) –USMA (3 runs) –Singapore (10 runs) Organization Types –18 Edge –19 Hierarchy

7. 7 Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

8. 8 Variables of Interest Info Sharing & Collaborative Behaviors Shared Information Quality of Information Shared Awareness Quality of Awareness Shared Understanding Quality of Understanding Task Performance Task Difficulty Measures of Merit Network Characteristics & Performance Individual & Team Characteristics Culture Allocation of Decision Rights Quality of Information Sources Patterns of Interaction Distribution of Information C2 Maturity Level Partially Controllable Controllable Legend

9. 9 Dependent Variables MOE = Quality of Awareness and Shared Awareness –Correctness (Authorized Correct IDs) –Timeliness (Person-Minutes with Correct IDs) –Accuracy rate (Correct IDs/Total IDs) Efficiency, Given Effectiveness –Productivity (Correct IDs/Total Actions; Correct IDs/Person- Minutes Available) –Speed (Time of Earliest Correct ID) Agility –Effectiveness over problem difficulty

10. 10 C2 Approach Independent Variables Hierarchy v. Edge –We expect Hierarchy to map to De-conflicted and Edge to map to a more mature level –Each run will be mapped to a point in the C2 Approach Space based on observed behaviors Rules of Interaction –Website access –Sharing permissions Initial Distribution of Factoids –Invariant in existing runs

11. 11 C2 Approach Intervening Variables Patterns of Interaction –Characteristic path length –Clustering coefficient –Connectedness Distribution of Information –The average number of unique facts to which each participant has access as a function of time

12. 12 Measures of C2 Effectiveness (MOCE) Quality of Information Position –Percentage of relevant facts for the assigned task that a participant can access as a function of time –Percentage of key facts for the assigned task that a participant can access as a function of time Extent of Shared Information –The average number of participants that have access to each fact as a function of time –The average number of participants that have access to each key fact as a function of time

13. 13 Intervening Behavioral Variables Activity over time (sharing, website posts, website pulls, ID attempts) –Sharing Peer-to-peer sharing Posting –Information Seeking Pulling –Identification Attempts

14. 14 Other Independent Variables IDs allowed ELICIT experience of the player Factoid set (problem difficulty) Translated factoids v. original Native Language (English v. Other) Communications media –Postcards –Chat Time available Degree of Education (Graduate, Undergraduate) Seniority (Rank) Subcultures (Military, Civilian, Special Forces, Civil Servants)

15. 15 Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

16. 16 Dependent Variables MOE = Quality of Awareness and Shared Awareness –Correctness (Authorized Correct IDs) –Timeliness (Person-Minutes with Correct IDs) –Accuracy rate (Correct IDs/Total IDs) Efficiency, Given Effectiveness –Productivity (Correct IDs/Total Actions; Correct IDs/Person- Minutes Available) –Speed (Time of Earliest Correct ID) Agility –Effectiveness over problem difficulty

17. 17 MOE: Correctness Fraction of Participants with Correct IDs Hierarchy Edge

18. 18 MOE: Correctness Fraction of Participants with Correct IDs Edge Better Than Hierarchy Significant at 0.95 level

19. 19 MOE: Correctness Fraction of Participants with Authorized Correct IDs Note: Half credit given for partially correct answers in authorized areas Hierarchy Edge

20. 20 MOE: Correctness Fraction of Authorized Participants with Correct ID Edge Better Than Hierarchy Significant at 0.95 level |t-Ratio| = 7.46

21. 21 Measuring Timeliness: Person-Minutes with Correct ID In the ELICIT scenario, operational value is associated both with the number of participants achieving the correct answer, as well as when they achieved it Person minutes with correct ID captures the proportion of possible situational understanding achieved by trial participants over the time of the trial –For each minute, each participant’s level of understanding is assessed by the score assigned to his/her most recent identification attempt –This value is summed over the duration of the trial to arrive at the person-minutes correct for each participant –Person-minutes correct for the trial is calculated by summing over all participants –A ratio of the person minutes correct for the trial to the total person minutes available in the trial adjusts for varying trial lengths

22. 22 MOE: Timeliness Fraction of Person-Minutes Correct Hierarchy Edge

23. 23 Edge Better Than Hierarchy Significant at 0.95 level MOE: Timeliness Fraction of Person-Minutes Correct

24. 24 MOE: Accuracy Rate Fraction of ID Attempts that are Correct Hierarchy Edge

25. 25 MOE: Accuracy Rate Fraction of ID Attempts that are Correct Edge Better Than Hierarchy Significant at 0.95 level

26. 26 Efficiency: Productivity (Actions) Correct ID / Total Actions Hierarchy Edge

27. 27 Efficiency: Productivity (Actions) Correct ID / Total Actions Difference not Significant at 0.95 level |t-Ratio| = 1.41

28. 28 Efficiency: Productivity (Person-Minutes) Correct IDs / Person-Minutes Available Hierarchy Edge

29. 29 Efficiency: Productivity (Person-Minutes) Correct IDs / Person-Minutes Available Edge Better Than Hierarchy Significant at 0.95 level

30. 30 Efficiency: Speed Time of Earliest Correct ID Hierarchy Edge

31. 31 Efficiency: Speed Time of Earliest Correct ID Difference not Significant at 0.95 level |t-Ratio| = 1.62

32. 32 Agility MOE: Timeliness over Problem Difficulty Edge Effectiveness is Degraded Less Than Hierarchy Fraction of Person-Minutes Correct Hierarchy Standard Factoid Sets Difficult Factoid Sets Edge Fraction of Person-Minutes Correct

33. 33 C2 Approach Intervening Variables Patterns of Interaction –Characteristic path length –Clustering coefficient –Connectedness Distribution of Information –The average number of unique facts to which each participant has access as a function of time

34. 34 Distribution of Information The cumulative number of unique facts to which each participant has access over time Time (minutes)

35. 35 Distribution of Information The cumulative number of unique facts to which each participant has access over time Time (minutes) Edge Better Than Hierarchy Significant at 0.95 level

36. 36 Measures of C2 Effectiveness Quality of Information Position –Percentage of relevant facts for the assigned task that a participant can access as a function of time –Percentage of key facts for the assigned task that a participant can access as a function of time Extent of Shared Information –The average number of participants that have access to each fact as a function of time –The average number of participants that have access to each key fact as a function of time

37. 37 MOCE: Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time Time (minutes)

38. 38 MOCE: Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time Time (minutes) Edge Better Than Hierarchy Significant at 0.95 level

39. 39 MOCE: Quality of Information Position Percentage of key facts for the assigned task that a participant can access as a function of time Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

40. 40 MOCE: Quality of Information Position Percentage of key facts for the assigned task that a participant can access as a function of time Time (minutes) Edge Better Than Hierarchy Significant at 0.95 level Key fact: Factoid Labeled as “Key” or “Key-Expertise”

41. 41 MOCE: Extent of Shared Information The average number of participants that have access to each fact as a function of time Time (minutes)

42. 42 MOCE: Extent of Shared Information The average number of participants that have access to each fact as a function of time Time (minutes) Edge Better Than Hierarchy Significant at 0.95 level

43. 43 MOCE: Extent of Shared Information The average number of participants that have access to each key fact as a function of time Key fact: Factoid Labeled as “Key” or “Key-Expertise” Time (minutes)

44. 44 MOCE: Extent of Shared Information The average number of participants that have access to each key fact as a function of time Key fact: Factoid Labeled as “Key” or “Key-Expertise” Time (minutes) Edge Better Than Hierarchy Significant at 0.95 level

45. 45 Intervening Behavioral Variables Activity over time (sharing, website posts, website pulls, ID) –Sharing Peer-to-peer Posting –Information Seeking Pulling –Identification

46. 46 Activity over Time Peer to Peer Sharing Time (minutes)

47. 47 Activity over Time Peer to Peer Sharing Time (minutes) Hierarchy Better Than Edge at 10 min No Significant Difference at 45 min (at 0.95 level)

48. 48 Activity over Time Posts Time (minutes)

49. 49 Activity over Time Posts Time (minutes) No Significant Difference at 0.95 level

50. 50 Activity over Time Information Seeking (Pulls) Time (minutes)

51. 51 Activity over Time Information Seeking (Pulls) Time (minutes) Edge Better Than Hierarchy Significant at 0.95 Level

52. 52 Activity over Time ID Attempts Time (minutes)

53. 53 Activity over Time ID Attempts Time (minutes) No Significant Difference at 0.95 level

54. 54 Summary of Findings Edge structures exhibit more mature behaviors than Hierarchical structures –Distribution of Information –Quality of Information Position –Extent of Shared Information –Information Seeking (Pulling) Edge structures are more effective than Hierarchical structures –Correctness –Timeliness –Accuracy Rate Edge structures are generally more efficient than Hierarchical structures –Productivity (Person-Minutes) –Productivity (Actions) – at 90% level –Speed – at 90% level Edge structures are more agile than Hierarchical structures –Effectiveness over Problem Difficulty

55. 55 Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

56. 56 Conclusion The data are consistent with the NATO C2 Maturity Model assumptions and hypotheses Additional analyses still underway –Extraction and analysis of network metrics –Bivariate and multivariate analyses