A Testbed for Studies of Team Cognition in the Cyber Security Domain Nancy J. Cooke Prashanth Rajivan Shankaranarayanan Venkatanarayanan Arizona State University 5 May 2010

Team Cognition Military, Cyber, and Medical Applications Cooke’s Background Background Education: Cognitive Psychology/Human Factors George Mason University, B.A. New Mexico State University, M.A., Ph.D. Positions Rice University New Mexico State University Arizona State University & Cognitive Engineering Research Institute Applied Experience: U.S Air Force, Navy, Army, NASA, NTSB, VA Section Editor, Human Factors USAF Scientific Advisory Board National Research Council Committee on Human Systems Integration Relevant Research Team Cognition Military, Cyber, and Medical Applications Communication Analysis Sponsors Air Force Office of Scientific Research Air Force Research Laboratory Office of Naval Research Army Research Office Leonard Wood Institute Veteran’s Administration – MWM VERC Metrics for Coordination and Collaboration

Overview MURI and ASU Team Team Cognition and Team Situation Awareness Other Team Testbeds CyberCog – New Testbed

MURI: Computer-aided Human Centric Cyber Situation Awareness DoD Multidisciplinary University Research Initiative (MURI) program project, funded through Army Research Office Two fundamental limitations of Cyber Situation Awareness (C-SA) Gap: human cognition < -- > C-SA tools Situation data exceeds “cognitive throughput” of human analysts “Blind spots” in views of cyber situation for existing C-SA tools (including auditing, vulnerability scanners, attack graph tools, intrusion detection systems, damage assessment tools, and forensics tools) Cyber-SA Vision Build data < -- > human decision links through innovations knowledge fusion cognitive automation artificial intelligence visual analytics Awareness-driven cyber defense vs. malware behavior dependent defense Automatic blind spot identification and monitoring techniques 　

MURI Partners Professor Peng Liu, Penn State University, Overall PI Professor Nancy Cooke, Arizona State University Professor Coty González, Carnegie Mellon University Professor Dave Hall, Penn State University Professor Sushil Jajodia, George Mason University Professor Mike McNeese, Penn State University Professor Peng Ning, NC State University Professor VS Subrahmanian, Univ. of Maryland Professor John Yen, Penn State University Professor Michael Young, NC State University 

ASU MURI Team Nancy J. Cooke Professor, Cognitive Science & Engineering College of Technology and Innovation Prashanth Rajivan Graduate Student Master’s in Computing Studies College of Technology Innovation Shankaranarayanan Venkatanarayanan Graduate Student Master’s in Computing Studies College of Technology and Innovation

Teams and Cognitive Tasks Team is unit of analysis = Heterogeneous and interdependent group of individuals (human or synthetic) who plan, decide, perceive, design, solve problems, and act as an integrated system. Cognitive activity at the team level= Team Cognition Improved team cognition  Improved team/system effectiveness Heterogeneous = differing backgrounds, differing perspectives on situation (surgery, basketball)

Some Instances of Failures of Team Cognition Unmanned Aerial Vehicles USS Vincennes shoots down Iranian airbus (1988) Challenger/Columbia accidents tied to poor organizational decision making (1986/2003) Response to 9/11 reveals communication breakdowns (2001) Katrina response lacked coordination (2005) Sago Mine disaster report cites poor command-and-control (2006) VA Tech communications substandard (2007) Friendly fire incidents Various health care mishaps attributed to poor teamwork

And some successes… Miracle on the Hudson Response to Fargo flooding

Interactive Team Cognition in a Nutshell Team interactions often in the form of explicit communications are the foundation of team cognition ASSUMPTIONS Team cognition is an activity; not a property or product Team cognition is inextricably tied to context Team cognition is best measured and studied when the team is the unit of analysis

US 2004 Olympic Basketball Team "We still have a couple of days, but I don't know where we are," replied USA head coach Larry Brown to a question Wednesday on where his team was in its preparations. "We have good moments and bad, but I've got a pretty good understanding of who needs to play. Now the job is to get an understanding of how we have to play." A team of experts does NOT make an expert team Collaborative skill is not additive

US 1980 Olympic Ice Hockey Team Herb Brooks and 20 young “no-names” won the 1980 Olympic Gold Medal in Ice Hockey An expert team made up of no-names…

Our UAV Testbed UAV-STE: Uninhabited Air Vehicle (ground control station) Synthetic Task Environment for research on team cognition (DURIP 1997; USAF funded) In our UAV STE three operators must coordinate over headsets in order to maneuver their UAV to take pictures of ground targets

Interdependence requires interaction, communication, & coordination DEMPC navigator, mission planner, plans route from target to target under constraints Three team members with inter- dependent tasks Payload Operator controls camera settings, takes photos, and monitors camera systems Air Vehicle Operator controls UAV airspeed, heading, and altitude and monitors air vehicle systems Interdependence requires interaction, communication, & coordination

Our MacroCog (Macro-Cognition Testbed) MacroCog Testbed Navy-funded lab for strategic planning and decision- making in the context of noncombatant evacuation operations 15

MacroCog Roles in Current Experiment Information Warfare Specialist Personnel Specialist: Military Equipment Specialist: Land/Sea Vehicles Experimenter 1 Experimenter 2 Personnel Specialist: Humanitarian Equipment Specialist: Air Vehicles 16

Example of Empirical Results on Team Cognition As teams acquire experience, performance improves, interactions improve, but not individual or collective knowledge 40-min missions Spring Break Individuals are trained to criterion prior to M1 Asymptotic team performance after 4 40-min missions (robust finding) Knowledge changes tend to occur in early learning (M1) and stabilize Process improves and communication becomes more standard over time

Team Situation Awareness A team’s coordinated perception and action in response to a change in the environment How can we exercise team SA in a testbed? How can we measure it? How can we intervene to improve it? Contrary to view that all team members need to “be on the same page”

What is Meant by Coordinated Perception and Action?

Measure of Team Situation Awareness Change is introduced (communication breakdown, enemy in area, storm) that will impact mission 2-3 team members are presented cues regarding change Team members need to perceive cues in a coordinated way (i.e., connect the dots) to identify the change Team members coordinate to take action relevant to the change (e.g., change altitude, communicate indirectly) Measure in terms of outcome and process – who on team was involved?

CyberCog Simulator Web based Simulator application for measuring individual interaction and team collaboration (e.g., team situation awareness) in a Cyber security analysis situation

CyberCogSimulator – System Overview This is an overview of the CyberCogSimulator setup. This above figure shows the resources available to the participants as well as the experimenter. It also shows the interactions, the information being exchanged between the participants, the real time logging of data onto the experimenters system and also the interaction between the experimenters system and MySQL database server. The Scenarios, Events ,Symptoms, Real-time user interaction information, Results for each session are all stored on to the Database server in an XML format. The Experimenter will be able to switch between multiple user screen through hardware control and also monitor the common screen shared between the users. 22

CyberCogSimulator – Components Cyber Security Analyst (User) Assigned a specific role such as Denial of Service (Dos) specialist, Malware specialist and Phishing specialist Understands the scenario given, use events and attack symptoms, collaborates with other participants to identify a potential attack or a combination of attacks The team reaches a common consensus on the type of attack and its corresponding events

CyberCogSimulator – Components Master controller and Evaluator Queries attack scenarios, events and symptoms from the database Distributes the events and symptoms to the participants Logs the interaction between participants at real time Evaluates and scores the participants findings with the expected results

CyberCogSimulator – Components Database server MySQL database server stores :- Attack Scenarios Events corresponding to attack scenarios including some false positives & noise events Attack Symptoms for each specialization (E.g., Dos, Malware , Phishing) identified The expected results, interaction (between participants ) logs and attack conclusion arrived at by each team for each session

User and Team Views Functions Data Legends User Screen Events Symptoms Match Broadcast Publish Unknown Common Screen Suspicious Events Submit The key data available and functions that can be performed at each of the screens(or views) are shown in this diagram. Match generates a tree like structure using the event and the symptom selected at screen. Broadcast , broadcasts the event selected to other team members when a matching symptom for the event is not available. On a confident match between a event and symptom, publish will help to post the event as suspicious at the common screen. Unknown pushes the selected event container on the same screen when the analyst is not able to identify the event with a symptom and the event is unknown to even other team members. When all the team members come to consensus, submit is to submit their findings. Legends Functions Data 26

CyberCog Simulator- Interaction This diagram show the event flow between views and between participants. It Shows how the application achieves interaction. 27

CyberCogSimulator- Architecture Microsoft IIS Web Services Dos Specialist POCO’s Intra/Internet Malware Specialist ADO.net The simulator application follows the MVC pattern – Model View Controller. In simple terms Model part of MVC is where the business logic and data access logic reside. The controller part of MVC is where the requests from users are tunneled to appropriate business logic. View of MVC is the actual output to be rendered to the user. So this diagram depicts the technologies used at each of the layers. The application controller and the views(asps) reside at the application server – Microsoft IIS. The business logic are available as POCO’s(Plain Old CLR objects) and web services. The ADO.net entity framework is used to access data from the database. MySQL is the database. Controller & View Tier Model Tier Phishing Specialist Client Tier Database 28

Conclusion There are current gaps and limitations in Cyber Situation Awareness Cyber situation awareness by teams involves the coordinated perception and action in the face of a change in the cyber situation CyberCog will allow the MURI team and others to better understand team-based cyber SA and to test algorithms and tools developed for improving it

Team Cognition Research Program UAS Field Data Testbeds: 1) UAS C2 2) Navy Strategic Planning Empirical Studies in Testbed Measures Situation Lang Gen Lang Comp Task Model ACT-R Model of Synthetic Teammate Theory Development Dynamical Systems Modeling