1. A Testbed for Studies of Team Cognition in the Cyber Security Domain
Nancy J. Cooke
Prashanth Rajivan
Shankaranarayanan Venkatanarayanan
Arizona State University
5 May 2010

2. Team Cognition Military, Cyber, and Medical Applications
Cooke’s Background
Background
Education: Cognitive Psychology/Human Factors
George Mason University, B.A.
New Mexico State University, M.A., Ph.D.
Positions
Rice University
New Mexico State University
Arizona State University &
Cognitive Engineering Research Institute
Applied Experience: U.S Air Force, Navy, Army, NASA, NTSB, VA
Section Editor, Human Factors
USAF Scientific Advisory Board
National Research Council Committee on Human Systems Integration
Relevant Research
Team Cognition Military, Cyber, and Medical Applications
Communication Analysis
Sponsors
Air Force Office of Scientific Research
Air Force Research Laboratory
Office of Naval Research
Army Research Office
Leonard Wood Institute
Veteran’s Administration – MWM VERC
Metrics for Coordination and Collaboration

3. Overview MURI and ASU Team Team Cognition and Team Situation Awareness
Other Team Testbeds
CyberCog – New Testbed

4. MURI: Computer-aided Human Centric Cyber Situation Awareness
DoD Multidisciplinary University Research Initiative (MURI) program project, funded through Army Research Office
Two fundamental limitations of Cyber Situation Awareness (C-SA)
Gap: human cognition < -- > C-SA tools
Situation data exceeds “cognitive throughput” of human analysts
“Blind spots” in views of cyber situation for existing C-SA tools (including auditing, vulnerability scanners, attack graph tools, intrusion detection systems, damage assessment tools, and forensics tools)
Cyber-SA Vision
Build data < -- > human decision links through innovations
knowledge fusion
cognitive automation
artificial intelligence
visual analytics
Awareness-driven cyber defense vs. malware behavior dependent defense
Automatic blind spot identification and monitoring techniques 　

5. MURI Partners Professor Peng Liu, Penn State University, Overall PI
Professor Nancy Cooke, Arizona State University
Professor Coty González, Carnegie Mellon University
Professor Dave Hall, Penn State University
Professor Sushil Jajodia, George Mason University
Professor Mike McNeese, Penn State University
Professor Peng Ning, NC State University
Professor VS Subrahmanian, Univ. of Maryland
Professor John Yen, Penn State University
Professor Michael Young, NC State University 

6. ASU MURI Team Nancy J. Cooke
Professor, Cognitive Science & Engineering
College of Technology and Innovation
Prashanth Rajivan
Graduate Student
Master’s in Computing Studies
College of Technology Innovation
Shankaranarayanan Venkatanarayanan
Graduate Student
Master’s in Computing Studies
College of Technology and Innovation

7. Teams and Cognitive Tasks
Team is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide, perceive, design, solve problems, and act as an integrated system.
Cognitive activity at the team level= Team Cognition
Improved team cognition  Improved team/system effectiveness
Heterogeneous = differing backgrounds, differing perspectives on situation (surgery, basketball)

8. Some Instances of Failures of Team Cognition
Unmanned Aerial Vehicles
USS Vincennes shoots down Iranian airbus (1988)
Challenger/Columbia accidents tied to poor organizational decision making (1986/2003)
Response to 9/11 reveals communication breakdowns (2001)
Katrina response lacked coordination (2005)
Sago Mine disaster report cites poor command-and-control (2006)
VA Tech communications substandard (2007)
Friendly fire incidents
Various health care mishaps attributed to poor teamwork

9. And some successes…
Miracle on the Hudson
Response to Fargo flooding

10. Interactive Team Cognition in a Nutshell
Team interactions often in the form of explicit communications are the foundation of team cognition
ASSUMPTIONS
Team cognition is an activity; not a property or product
Team cognition is inextricably tied to context
Team cognition is best measured and studied when the team is the unit of analysis

11. US 2004 Olympic Basketball Team
"We still have a couple of days, but I don't know where we are," replied USA head coach Larry Brown to a question Wednesday on where his team was in its preparations. "We have good moments and bad, but I've got a pretty good understanding of who needs to play. Now the job is to get an understanding of how we have to play."
A team of experts does NOT make an expert team
Collaborative skill is not additive

12. US 1980 Olympic Ice Hockey Team
Herb Brooks and 20 young “no-names” won the 1980 Olympic Gold Medal in Ice Hockey
An expert team made up of no-names…

13. Our UAV Testbed
UAV-STE: Uninhabited Air Vehicle (ground control station) Synthetic Task Environment for research on team cognition (DURIP 1997; USAF funded)
In our UAV STE three operators must coordinate over headsets in order to maneuver their UAV to take pictures of ground targets

14. Interdependence requires interaction, communication, & coordination
DEMPC
navigator, mission planner, plans route from target to target under constraints
Three team members with inter-
dependent tasks
Payload Operator controls camera settings, takes photos, and monitors camera systems
Air Vehicle Operator controls UAV airspeed, heading, and altitude and monitors air vehicle systems
Interdependence requires interaction, communication, & coordination

15. Our MacroCog (Macro-Cognition Testbed)
MacroCog Testbed
Navy-funded lab for strategic planning and decision- making in the context of noncombatant evacuation operations 15

16. MacroCog Roles in Current Experiment
Information Warfare Specialist
Personnel Specialist: Military
Equipment Specialist:
Land/Sea Vehicles
Experimenter 1
Experimenter 2
Personnel Specialist: Humanitarian
Equipment Specialist: Air Vehicles 16

17. Example of Empirical Results on Team Cognition As teams acquire experience, performance improves, interactions improve, but not individual or collective knowledge
40-min missions
Spring Break
Individuals are trained to criterion prior to M1
Asymptotic team performance after 4 40-min missions (robust finding)
Knowledge changes tend to occur in early learning (M1) and stabilize
Process improves and communication becomes more standard over time

18. Team Situation Awareness
A team’s coordinated perception and action in response to a change in the environment
How can we exercise team SA in a testbed?
How can we measure it?
How can we intervene to improve it?
Contrary to view that all team members need to “be on the same page”

19. What is Meant by Coordinated Perception and Action?

20. Measure of Team Situation Awareness
Change is introduced (communication breakdown, enemy in area, storm) that will impact mission
2-3 team members are presented cues regarding change
Team members need to perceive cues in a coordinated way (i.e., connect the dots) to identify the change
Team members coordinate to take action relevant to the change (e.g., change altitude, communicate indirectly)
Measure in terms of outcome and process – who on team was involved?

21. CyberCog Simulator
Web based Simulator application for measuring individual interaction and team collaboration (e.g., team situation awareness) in a Cyber security analysis situation

22. CyberCogSimulator – System Overview
This is an overview of the CyberCogSimulator setup. This above figure shows the resources available to the participants as well as the experimenter. It also shows the interactions, the information being exchanged between the participants, the real time logging of data onto the experimenters system and also the interaction between the experimenters system and MySQL database server.
The Scenarios, Events ,Symptoms, Real-time user interaction information, Results for each session are all stored on to the Database server in an XML format. The Experimenter will be able to switch between multiple user screen through hardware control and also monitor the common screen shared between the users. 22

23. CyberCogSimulator – Components
Cyber Security Analyst (User)
Assigned a specific role such as Denial of Service (Dos) specialist, Malware specialist and Phishing specialist
Understands the scenario given, use events and attack symptoms, collaborates with other participants to identify a potential attack or a combination of attacks
The team reaches a common consensus on the type of attack and its corresponding events

24. CyberCogSimulator – Components
Master controller and Evaluator
Queries attack scenarios, events and symptoms from the database
Distributes the events and symptoms to the participants
Logs the interaction between participants at real time
Evaluates and scores the participants findings with the expected results

25. CyberCogSimulator – Components
Database server
MySQL database server stores :-
Attack Scenarios
Events corresponding to attack scenarios including some false positives & noise events
Attack Symptoms for each specialization (E.g., Dos, Malware , Phishing) identified
The expected results, interaction (between participants ) logs and attack conclusion arrived at by each team for each session

26. User and Team Views Functions Data Legends
User Screen
Events
Symptoms
Match
Broadcast
Publish
Unknown
Common Screen
Suspicious Events
Submit
The key data available and functions that can be performed at each of the screens(or views) are shown in this diagram. Match generates a tree like structure using the event and the symptom selected at screen. Broadcast , broadcasts the event selected to other team members when a matching symptom for the event is not available. On a confident match between a event and symptom, publish will help to post the event as suspicious at the common screen. Unknown pushes the selected event container on the same screen when the analyst is not able to identify the event with a symptom and the event is unknown to even other team members. When all the team members come to consensus, submit is to submit their findings.
Legends
Functions
Data 26

27. CyberCog Simulator- Interaction
This diagram show the event flow between views and between participants. It Shows how the application achieves interaction. 27

28. CyberCogSimulator- Architecture
Microsoft IIS
Web Services
Dos Specialist
POCO’s
Intra/Internet
Malware Specialist
ADO.net
The simulator application follows the MVC pattern – Model View Controller. In simple terms Model part of MVC is where the business logic and data access logic reside. The controller part of MVC is where the requests from users are tunneled to appropriate business logic. View of MVC is the actual output to be rendered to the user. So this diagram depicts the technologies used at each of the layers. The application controller and the views(asps) reside at the application server – Microsoft IIS. The business logic are available as POCO’s(Plain Old CLR objects) and web services. The ADO.net entity framework is used to access data from the database. MySQL is the database.
Controller &
View Tier
Model Tier
Phishing Specialist
Client Tier
Database 28

29. Conclusion
There are current gaps and limitations in Cyber Situation Awareness
Cyber situation awareness by teams involves the coordinated perception and action in the face of a change in the cyber situation
CyberCog will allow the MURI team and others to better understand team-based cyber SA and to test algorithms and tools developed for improving it

30. Team Cognition Research Program
UAS Field Data
Testbeds:
1) UAS C2
2) Navy Strategic Planning
Empirical Studies in Testbed
Measures
Situation
Lang Gen
Lang Comp
Task Model
ACT-R Model of Synthetic Teammate
Theory Development
Dynamical Systems Modeling