C2 Maturity Model Experimental Validation Statistical Analyses of ELICIT Experimentation Data Dr. David S. Alberts

Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

Background SAS-065 has developed a C2 Maturity Model for NATO Network Enabled Capability SAS-065 is engaged in a variety of activities to “validate” this model Case studies: Analysis of historical complex endeavors to establish construct validity Analysis of Experimentation Data to test a set of maturity model hypotheses ELICIT is an experimentation platform that instruments the actions of a group of participants engaged in a situational awareness problem ELICIT experiments have explored differences between “edge” and “hierarchical” organizations

ELICIT Scenario The goal of each set of participants is to build situational awareness and identify the who, what, when, and where of a pending attack Participants can share factoids directly with each other or post factoids to websites Participants build awareness by gathering and analyzing factoids and interacting with one another No one is given sufficient information to solve their assigned problem without receiving information from others The receiving, sharing, and posting of factoids and the nature of the interactions between and among participants can be constrained C2 approach for this series of experiments were designated prior to the start of the run as Hierarchy or Edge Hierarchy Edge

Hypotheses “Hierarchical” organizations as instantiated in ELICIT experiments correspond to De-Conflicted C2 in the NATO C2 Maturity Model “Edge” organizations as instantiated in ELICIT experiments correspond to a higher level of maturity in the NATO C2 Maturity Model Hypotheses: Edge organizations exhibit the behaviors associated with Coordinated and Collaborative levels of C2 Maturity Hierarchical organizations exhibit the behaviors associated with the De-conflicted level of C2 Maturity Edge organizations exhibit higher levels of effectiveness and efficiency than Hierarchical organizations

ELICIT Data Set Includes data from 37 ELICIT experimentation trials Venues Boston Univ. (2 runs) NPS (16 runs) Portugal (6 runs) USMA (3 runs) Singapore (10 runs) Organization Types 18 Edge 19 Hierarchy

Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

Partially Controllable Variables of Interest C2 Maturity Level Allocation of Decision Rights Patterns of Interaction Network Characteristics & Performance Distribution of Information Shared Information Shared Awareness Shared Understanding Info Sharing & Collaborative Behaviors Task Performance Quality of Information Quality of Awareness Quality of Understanding Measures of Merit Task Difficulty Individual & Team Characteristics Legend Controllable Quality of Information Sources Partially Controllable Culture

Dependent Variables MOE = Quality of Awareness and Shared Awareness Correctness (Authorized Correct IDs) Timeliness (Person-Minutes with Correct IDs) Accuracy rate (Correct IDs/Total IDs) Efficiency, Given Effectiveness Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available) Speed (Time of Earliest Correct ID) Agility Effectiveness over problem difficulty

C2 Approach Independent Variables Hierarchy v. Edge We expect Hierarchy to map to De-conflicted and Edge to map to a more mature level Each run will be mapped to a point in the C2 Approach Space based on observed behaviors Rules of Interaction Website access Sharing permissions Initial Distribution of Factoids Invariant in existing runs

C2 Approach Intervening Variables Patterns of Interaction Characteristic path length Clustering coefficient Connectedness Distribution of Information The average number of unique facts to which each participant has access as a function of time

Measures of C2 Effectiveness (MOCE) Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time Percentage of key facts for the assigned task that a participant can access as a function of time Extent of Shared Information The average number of participants that have access to each fact as a function of time The average number of participants that have access to each key fact as a function of time

Intervening Behavioral Variables Activity over time (sharing, website posts, website pulls, ID attempts) Sharing Peer-to-peer sharing Posting Information Seeking Pulling Identification Attempts

Other Independent Variables IDs allowed ELICIT experience of the player Factoid set (problem difficulty) Translated factoids v. original Native Language (English v. Other) Communications media Postcards Chat Time available Degree of Education (Graduate, Undergraduate) Seniority (Rank) Subcultures (Military, Civilian, Special Forces, Civil Servants)

Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

Dependent Variables MOE = Quality of Awareness and Shared Awareness Correctness (Authorized Correct IDs) Timeliness (Person-Minutes with Correct IDs) Accuracy rate (Correct IDs/Total IDs) Efficiency, Given Effectiveness Productivity (Correct IDs/Total Actions; Correct IDs/Person-Minutes Available) Speed (Time of Earliest Correct ID) Agility Effectiveness over problem difficulty

MOE: Correctness Fraction of Participants with Correct IDs Edge Hierarchy Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards.

MOE: Correctness Fraction of Participants with Correct IDs Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards. Edge Better Than Hierarchy Significant at 0.95 level

MOE: Correctness Fraction of Participants with Authorized Correct IDs Edge Hierarchy Note: Half credit given for partially correct answers in authorized areas

MOE: Correctness Fraction of Authorized Participants with Correct ID Edge Better Than Hierarchy Significant at 0.95 level |t-Ratio| = 7.46

Measuring Timeliness: Person-Minutes with Correct ID In the ELICIT scenario, operational value is associated both with the number of participants achieving the correct answer, as well as when they achieved it Person minutes with correct ID captures the proportion of possible situational understanding achieved by trial participants over the time of the trial For each minute, each participant’s level of understanding is assessed by the score assigned to his/her most recent identification attempt This value is summed over the duration of the trial to arrive at the person-minutes correct for each participant Person-minutes correct for the trial is calculated by summing over all participants A ratio of the person minutes correct for the trial to the total person minutes available in the trial adjusts for varying trial lengths

MOE: Timeliness Fraction of Person-Minutes Correct Edge Hierarchy

MOE: Timeliness Fraction of Person-Minutes Correct Edge Better Than Hierarchy Significant at 0.95 level

MOE: Accuracy Rate Fraction of ID Attempts that are Correct Edge Hierarchy Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards.

MOE: Accuracy Rate Fraction of ID Attempts that are Correct Note: a more detailed search of the 94% correct value in the Edge runs showed that 16 correct guesses occurred in the run Edge_with_Chat_1A.log which uses a chat capability. The other high percentages occur in Edge_with_Chat_5A.log, which uses chat, and some of the NPS runs which may have used postcards. Edge Better Than Hierarchy Significant at 0.95 level

Efficiency: Productivity (Actions) Correct ID / Total Actions Edge Hierarchy Note: Hierarchy_with_Chat_6B.log

Efficiency: Productivity (Actions) Correct ID / Total Actions Note: Hierarchy_with_Chat_6B.log Difference not Significant at 0.95 level |t-Ratio| = 1.41

Efficiency: Productivity (Person-Minutes) Correct IDs / Person-Minutes Available Edge Hierarchy

Edge Better Than Hierarchy Significant at 0.95 level Efficiency: Productivity (Person-Minutes) Correct IDs / Person-Minutes Available Edge Better Than Hierarchy Significant at 0.95 level

Efficiency: Speed Time of Earliest Correct ID Edge Hierarchy

Efficiency: Speed Time of Earliest Correct ID Difference not Significant at 0.95 level |t-Ratio| = 1.62

Agility MOE: Timeliness over Problem Difficulty Fraction of Person-Minutes Correct Fraction of Person-Minutes Correct Difficult Factoid Sets Fraction of Person-Minutes Correct Fraction of Person-Minutes Correct Standard Factoid Sets Edge Hierarchy Edge Effectiveness is Degraded Less Than Hierarchy

C2 Approach Intervening Variables Patterns of Interaction Characteristic path length Clustering coefficient Connectedness Distribution of Information The average number of unique facts to which each participant has access as a function of time

Distribution of Information The cumulative number of unique facts to which each participant has access over time Time (minutes)

Edge Better Than Hierarchy Significant at 0.95 level Distribution of Information The cumulative number of unique facts to which each participant has access over time Edge Better Than Hierarchy Significant at 0.95 level Time (minutes)

Measures of C2 Effectiveness Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time Percentage of key facts for the assigned task that a participant can access as a function of time Extent of Shared Information The average number of participants that have access to each fact as a function of time The average number of participants that have access to each key fact as a function of time

MOCE: Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time Time (minutes)

Edge Better Than Hierarchy Significant at 0.95 level MOCE: Quality of Information Position Percentage of relevant facts for the assigned task that a participant can access as a function of time Edge Better Than Hierarchy Significant at 0.95 level Time (minutes)

MOCE: Quality of Information Position Percentage of key facts for the assigned task that a participant can access as a function of time Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

Edge Better Than Hierarchy Significant at 0.95 level MOCE: Quality of Information Position Percentage of key facts for the assigned task that a participant can access as a function of time Edge Better Than Hierarchy Significant at 0.95 level Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

MOCE: Extent of Shared Information The average number of participants that have access to each fact as a function of time Time (minutes)

Edge Better Than Hierarchy Significant at 0.95 level MOCE: Extent of Shared Information The average number of participants that have access to each fact as a function of time Edge Better Than Hierarchy Significant at 0.95 level Time (minutes)

MOCE: Extent of Shared Information The average number of participants that have access to each key fact as a function of time Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

Edge Better Than Hierarchy Significant at 0.95 level MOCE: Extent of Shared Information The average number of participants that have access to each key fact as a function of time Edge Better Than Hierarchy Significant at 0.95 level Time (minutes) Key fact: Factoid Labeled as “Key” or “Key-Expertise”

Intervening Behavioral Variables Activity over time (sharing, website posts, website pulls, ID) Sharing Peer-to-peer Posting Information Seeking Pulling Identification

Activity over Time Peer to Peer Sharing Time (minutes)

Activity over Time Peer to Peer Sharing Hierarchy Better Than Edge at 10 min No Significant Difference at 45 min (at 0.95 level) Time (minutes)

Activity over Time Posts Time (minutes)

Activity over Time Posts No Significant Difference at 0.95 level Time (minutes)

Activity over Time Information Seeking (Pulls) Time (minutes)

Activity over Time Information Seeking (Pulls) Edge Better Than Hierarchy Significant at 0.95 Level Time (minutes)

Activity over Time ID Attempts Time (minutes)

Activity over Time ID Attempts No Significant Difference at 0.95 level Time (minutes)

Summary of Findings Edge structures exhibit more mature behaviors than Hierarchical structures Distribution of Information Quality of Information Position Extent of Shared Information Information Seeking (Pulling) Edge structures are more effective than Hierarchical structures Correctness Timeliness Accuracy Rate Edge structures are generally more efficient than Hierarchical structures Productivity (Person-Minutes) Productivity (Actions) – at 90% level Speed – at 90% level Edge structures are more agile than Hierarchical structures Effectiveness over Problem Difficulty

Agenda Background Available Data Analysis Plan Results of Analyses Conclusions

Conclusion The data are consistent with the NATO C2 Maturity Model assumptions and hypotheses Additional analyses still underway Extraction and analysis of network metrics Bivariate and multivariate analyses