1. C2 Maturity Model Experimental Validation
Statistical Analyses of ELICIT Experimentation Data
Dr. David S. Alberts

2. Agenda Background Available Data Analysis Plan Results of Analyses
Conclusions

3. Background
SAS-065 has developed a C2 Maturity Model for NATO Network Enabled Capability
SAS-065 is engaged in a variety of activities to “validate” this model
Case studies: Analysis of historical complex endeavors to establish construct validity
Analysis of Experimentation Data to test a set of maturity model hypotheses
ELICIT is an experimentation platform that instruments the actions of a group of participants engaged in a situational awareness problem
ELICIT experiments have explored differences between “edge” and “hierarchical” organizations

4. ELICIT Scenario
The goal of each set of participants is to build situational awareness and identify the who, what, when, and where of a pending attack
Participants can share factoids directly with each other or post factoids to websites
Participants build awareness by gathering and analyzing factoids and interacting with one another
No one is given sufficient information to solve their assigned problem without receiving information from others
The receiving, sharing, and posting of factoids and the nature of the interactions between and among participants can be constrained
C2 approach for this series of experiments were designated prior to the start of the run as Hierarchy or Edge
Hierarchy
Edge

5. Hypotheses
“Hierarchical” organizations as instantiated in ELICIT experiments correspond to De-Conflicted C2 in the NATO C2 Maturity Model
“Edge” organizations as instantiated in ELICIT experiments correspond to a higher level of maturity in the NATO C2 Maturity Model
Hypotheses:
Edge organizations exhibit the behaviors associated with Coordinated and Collaborative levels of C2 Maturity
Hierarchical organizations exhibit the behaviors associated with the De-conflicted level of C2 Maturity
Edge organizations exhibit higher levels of effectiveness and efficiency than Hierarchical organizations

6. ELICIT Data Set Includes data from 37 ELICIT experimentation trials
Venues
Boston Univ. (2 runs)
NPS (16 runs)
Portugal (6 runs)
USMA (3 runs)
Singapore (10 runs)
Organization Types
18 Edge
19 Hierarchy

7. Agenda Background Available Data Analysis Plan Results of Analyses
Conclusions

8. Partially Controllable
Variables of Interest
C2 Maturity Level
Allocation of Decision Rights
Patterns of Interaction
Network
Characteristics
& Performance
Distribution of Information
Shared
Information
Shared
Awareness
Shared
Understanding
Info Sharing &
Collaborative
Behaviors
Task
Performance
Quality of
Information
Quality of
Awareness
Quality of
Understanding
Measures of Merit
Task
Difficulty
Individual
& Team
Characteristics
Legend
Controllable
Quality of
Information
Sources
Partially Controllable
Culture

9. Dependent Variables MOE = Quality of Awareness and Shared Awareness
Correctness (Authorized Correct IDs)
Timeliness (Person-Minutes with Correct IDs)
Accuracy rate (Correct IDs/Total IDs)
Efficiency, Given Effectiveness
Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available)
Speed (Time of Earliest Correct ID)
Agility
Effectiveness over problem difficulty

10. C2 Approach Independent Variables
Hierarchy v. Edge
We expect Hierarchy to map to De-conflicted and Edge to map to a more mature level
Each run will be mapped to a point in the C2 Approach Space based on observed behaviors
Rules of Interaction
Website access
Sharing permissions
Initial Distribution of Factoids
Invariant in existing runs

11. C2 Approach Intervening Variables
Patterns of Interaction
Characteristic path length
Clustering coefficient
Connectedness
Distribution of Information
The average number of unique facts to which each participant has access as a function of time

12. Measures of C2 Effectiveness (MOCE)
Quality of Information Position
Percentage of relevant facts for the assigned task that a participant can access as a function of time
Percentage of key facts for the assigned task that a participant can access as a function of time
Extent of Shared Information
The average number of participants that have access to each fact as a function of time
The average number of participants that have access to each key fact as a function of time

13. Intervening Behavioral Variables
Activity over time (sharing, website posts, website pulls, ID attempts)
Sharing
Peer-to-peer sharing
Posting
Information Seeking
Pulling
Identification Attempts

14. Other Independent Variables
IDs allowed
ELICIT experience of the player
Factoid set (problem difficulty)
Translated factoids v. original
Native Language (English v. Other)
Communications media
Postcards
Chat
Time available
Degree of Education (Graduate, Undergraduate)
Seniority (Rank)
Subcultures (Military, Civilian, Special Forces, Civil Servants)

15. Agenda Background Available Data Analysis Plan Results of Analyses
Conclusions

16. Dependent Variables MOE = Quality of Awareness and Shared Awareness
Correctness (Authorized Correct IDs)
Timeliness (Person-Minutes with Correct IDs)
Accuracy rate (Correct IDs/Total IDs)
Efficiency, Given Effectiveness
Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available)
Speed (Time of Earliest Correct ID)
Agility
Effectiveness over problem difficulty

17. MOE: Correctness Fraction of Participants with Correct IDs
Edge
Hierarchy
Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards.

18. MOE: Correctness Fraction of Participants with Correct IDs
Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards.
Edge Better Than Hierarchy Significant at 0.95 level

19. MOE: Correctness Fraction of Participants with Authorized Correct IDs
Edge
Hierarchy
Note: Half credit given for partially correct answers in authorized areas

20. MOE: Correctness Fraction of Authorized Participants with Correct ID
Edge Better Than Hierarchy Significant at 0.95 level
|t-Ratio| = 7.46

21. Measuring Timeliness: Person-Minutes with Correct ID
In the ELICIT scenario, operational value is associated both with the number of participants achieving the correct answer, as well as when they achieved it
Person minutes with correct ID captures the proportion of possible situational understanding achieved by trial participants over the time of the trial
For each minute, each participant’s level of understanding is assessed by the score assigned to his/her most recent identification attempt
This value is summed over the duration of the trial to arrive at the person-minutes correct for each participant
Person-minutes correct for the trial is calculated by summing over all participants
A ratio of the person minutes correct for the trial to the total person minutes available in the trial adjusts for varying trial lengths

22. MOE: Timeliness Fraction of Person-Minutes Correct
Edge
Hierarchy

23. MOE: Timeliness Fraction of Person-Minutes Correct
Edge Better Than Hierarchy Significant at 0.95 level

24. MOE: Accuracy Rate Fraction of ID Attempts that are Correct
Edge
Hierarchy
Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards.

25. MOE: Accuracy Rate Fraction of ID Attempts that are Correct
Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards.
Edge Better Than Hierarchy Significant at 0.95 level

26. Efficiency: Productivity (Actions) Correct ID / Total Actions
Edge
Hierarchy
Note: Hierarchy_with_Chat_6B.log

27. Efficiency: Productivity (Actions) Correct ID / Total Actions
Note: Hierarchy_with_Chat_6B.log
Difference not Significant at 0.95 level
|t-Ratio| = 1.41

28. Efficiency: Productivity (Person-Minutes) Correct IDs / Person-Minutes Available
Edge
Hierarchy

29. Edge Better Than Hierarchy Significant at 0.95 level
Efficiency: Productivity (Person-Minutes) Correct IDs / Person-Minutes Available
Edge Better Than Hierarchy Significant at 0.95 level

30. Efficiency: Speed Time of Earliest Correct ID
Edge
Hierarchy

31. Efficiency: Speed Time of Earliest Correct ID
Difference not Significant at 0.95 level
|t-Ratio| = 1.62

32. Agility MOE: Timeliness over Problem Difficulty
Fraction of Person-Minutes Correct
Fraction of Person-Minutes Correct
Difficult Factoid Sets
Fraction of Person-Minutes Correct
Fraction of Person-Minutes Correct
Standard Factoid Sets
Edge
Hierarchy
Edge Effectiveness is Degraded Less Than Hierarchy

33. C2 Approach Intervening Variables
Patterns of Interaction
Characteristic path length
Clustering coefficient
Connectedness
Distribution of Information
The average number of unique facts to which each participant has access as a function of time

34. Distribution of Information The cumulative number of unique facts to which each participant has access over time
Time (minutes)

35. Edge Better Than Hierarchy Significant at 0.95 level
Distribution of Information The cumulative number of unique facts to which each participant has access over time
Edge Better Than Hierarchy Significant at 0.95 level
Time (minutes)

36. Measures of C2 Effectiveness
Quality of Information Position
Percentage of relevant facts for the assigned task that a participant can access as a function of time
Percentage of key facts for the assigned task that a participant can access as a function of time
Extent of Shared Information
The average number of participants that have access to each fact as a function of time
The average number of participants that have access to each key fact as a function of time

37. MOCE: Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time
Time (minutes)

38. Edge Better Than Hierarchy Significant at 0.95 level
MOCE: Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time
Edge Better Than Hierarchy Significant at 0.95 level
Time (minutes)

39. MOCE: Quality of Information Position Percentage of key facts for the assigned task that a participant can access as a function of time
Time (minutes)
Key fact: Factoid Labeled as “Key” or “Key-Expertise”

40. Edge Better Than Hierarchy Significant at 0.95 level
MOCE: Quality of Information Position Percentage of key facts for the assigned task that a participant can access as a function of time
Edge Better Than Hierarchy Significant at 0.95 level
Time (minutes)
Key fact: Factoid Labeled as “Key” or “Key-Expertise”

41. MOCE: Extent of Shared Information The average number of participants that have access to each fact as a function of time
Time (minutes)

42. Edge Better Than Hierarchy Significant at 0.95 level
MOCE: Extent of Shared Information The average number of participants that have access to each fact as a function of time
Edge Better Than Hierarchy Significant at 0.95 level
Time (minutes)

43. MOCE: Extent of Shared Information The average number of participants that have access to each key fact as a function of time
Time (minutes)
Key fact: Factoid Labeled as “Key” or “Key-Expertise”

44. Edge Better Than Hierarchy Significant at 0.95 level
MOCE: Extent of Shared Information The average number of participants that have access to each key fact as a function of time
Edge Better Than Hierarchy Significant at 0.95 level
Time (minutes)
Key fact: Factoid Labeled as “Key” or “Key-Expertise”

45. Intervening Behavioral Variables
Activity over time (sharing, website posts, website pulls, ID)
Sharing
Peer-to-peer
Posting
Information Seeking
Pulling
Identification

46. Activity over Time Peer to Peer Sharing
Time (minutes)

47. Activity over Time Peer to Peer Sharing
Hierarchy Better Than Edge at 10 min No Significant Difference at 45 min (at 0.95 level)
Time (minutes)

48. Activity over Time Posts
Time (minutes)

49. Activity over Time Posts
No Significant Difference at 0.95 level
Time (minutes)

50. Activity over Time Information Seeking (Pulls)
Time (minutes)

51. Activity over Time Information Seeking (Pulls)
Edge Better Than Hierarchy Significant at 0.95 Level
Time (minutes)

52. Activity over Time ID Attempts
Time (minutes)

53. Activity over Time ID Attempts
No Significant Difference
at 0.95 level
Time (minutes)

54. Summary of Findings
Edge structures exhibit more mature behaviors than Hierarchical structures
Distribution of Information
Quality of Information Position
Extent of Shared Information
Information Seeking (Pulling)
Edge structures are more effective than Hierarchical structures
Correctness
Timeliness
Accuracy Rate
Edge structures are generally more efficient than Hierarchical structures
Productivity (Person-Minutes)
Productivity (Actions) – at 90% level
Speed – at 90% level
Edge structures are more agile than Hierarchical structures
Effectiveness over Problem Difficulty

55. Agenda Background Available Data Analysis Plan Results of Analyses
Conclusions

56. Conclusion
The data are consistent with the NATO C2 Maturity Model assumptions and hypotheses
Additional analyses still underway
Extraction and analysis of network metrics
Bivariate and multivariate analyses