1. 1 MURI: Computer-aided Human Centric Cyber Situation Awareness Peng Liu Professor & Director, The LIONS Center Pennsylvania State University ARO Cyber Situation Awareness MURI

2. Team Peng Liu, Professor and Director, Penn State Center for Cyber-Security, Information Privacy and Trust Massimiliano Albanese, Assistant Professor, GMU Nancy Cooke, Professor and Science Director, Arizona State Cognitive Engineering Research Institute Coty González, Associate Research Professor and Director, CMU Dynamic Decision Making Lab Dave Hall, Professor and Dean, Penn State College of IST Christopher Healey, Associate Professor, NC State 2 Sushil Jajodia, University Professor and Director, George Mason Univ. Center for Secure Information Systems Mike McNeese, Professor and Associate Dean, Penn State College of IST Peng Ning (on leave), Professor, NCSU Douglas Reeves, Professor and Interim Assistant Dean for COE Graduate Programs, NCSU VS Subrahmanian, Professor and past Director, U. of Maryland Institute for Advanced Computer Studies John Yen, University Professor and Director, Intelligent Agents Lab # of post docs: 5 # of graduate students: 18

3. 3 ARO MURI: Computer-aided Human Centric Cyber Situation Awareness PSU, ASU, CMU, GMU, NCSU, UMD Contact: Peng Liu, Tel. 814-863-0641, E-Mail: pliu@ist.psu.edupliu@ist.psu.edu Objectives: Improve Cyber SA through: Cyber SA specific cognition models. Cognition-friendly tools and analytics that fill the gap between the sensor side and the analyst side of cyber SA. Cross-layer situation knowledge integration. DoD Benefit: Significantly improved capabilities in gaining cyber SA in face of cyber attacks. Significantly improved job performance of analysts. Scientific/Technical Approach Take a holistic approach to integrate the “human cognition” aspects and the “cyber tools” aspects of cyber SA. Leverage cognition models to develop human cognition-friendly SA techniques, tools, and analytics. Accomplishments Year 4: See slide 5 Challenges Understanding the mental processes of analysts Team integration

4. Security Analysts Computer network Multi-Sensory Human Computer Interaction Enterprise Model Activity Logs IDS reports Vulnerabilities Cognitive Models & Decision Aids Instance Based Learning Models Simulation Measures of SA & Shared SA Data Conditioning Association & Correlation Automated Reasoning Tools R-CAST Plan-based narratives Graphical models Uncertainty analysis Information Aggregation & Fusion Transaction Graph methods Damage assessment Computer network Real World Test- bed 4

5. 5 Year 4 accomplishments Pub: -- 40 (13 journals, 24 conf., 3 chapters) -- 4 PhD thesis, 2 MS thesis -- 9 presentations Tools: -- ARSCA -- MetaSymploit -- NETS simulator -- DEXTAR -- Patrol -- Switchwall -- NSDMiner -- CyberCog -- PASS -- CAULDRON -- etc. Technology transitions: -- See slides later on Research: -- Major breakthroughs made -- See individual presentations Deep collaboration with ARL: -- 11 ARL security analysts -- 5 researchers at ARL -- Yen as summer faculty fellow -- 3 papers plus several in preparation

6. Year 4 accomplishments (cont’d) 6 Best Paper Award, SECRYPT 2013, “An Efficient Approach to Assessing the Risk of Zero-Day Vulnerabilities” by M. Albanese, S. Jajodia, A. Singhal, and L. Wang. HFES 2013 Alphonse Chapanis Award for best student paper, Prashanth Rajivan Sushil Jajodia, IEEE Fellow, January 2013. VAST Challenge 2013 Honorable Mention - Noteworthy Collaborative Analysis Strategy, by C. Zhong, M. Zhao, J. Xu, and G. Xiao, Leveraging "Visualization Functions" in Hypothesis-based Collaboration on Cyber Analysis Grace Hopper Scholarship 2013: Chen Zhong

7. What has happened? What is the impact? Why did it happen? What should I do? Security Analysts Computer networks (e.g., GIG) Sensors, probes Cyber Operations for Mission Assurance 7

8. Cyber Situation Awareness 8 What has happened? What is the impact? Why did it happen? Core Cyber SA Enabler What should I do?

9. Cyber SA Info Processing Box The Network Attacks Data Sources (feeds) Depicted Situation Ground Truth (estimates) Compare Job Performance 9

10. Why Research is Needed? 10 20+ CNDSPs*, whose operations are relying on human analysts, face critical challenges: 1. Job performance is unstable 2. Hard to get the big picture: walls between functional domains 3. Better analytics and tools are needed to improve job performance * In the commercial world, similar issues exist.

11. State of the Art: Big Gap Exists 11 Ability to create problem- solving workflows To see big picture To manage uncertainty To reason albeit incomplete/noisy knowledge To quickly locate needles in haystacks To do strategic planning To predict … Vulnerability scan Event logging Traffic classifying Intrusion detection Alert correlation Signature gen. Taint analysis Back tracking Integrity check Static analysis Bug finding Attack graphs Symbolic execution Sandbox VM monitors … Current tools: BIG GAP Desired cyber SA capabilities:

12. Scientific Objectives 12 Develop a deep understanding on: 1.Why the job performance between expert and rookie analysts is so different? How to bridge the job performance gap? 2.Why many tools cannot effectively improve job performance? 3.What models, tools and analytics are needed to effectively boost job performance? Develop a new paradigm of cyber SA system design, implementation, and evaluation. Tackle the scientific barriers on next slide.

13. Scientific Barriers 13 A.Massive amounts of sensed info vs. poorly used by analysts B.Silicon-speed info sensing vs. neuron-speed human cognition C.Stovepiped sensing vs. the need for "big picture awareness" D.Knowledge of “us” E.Lack of ground-truth vs. the need for scientifically sound models F.Unknown adversary intent vs. publicly-known vulnerability categories

14. Potential Scientific Advances 14 Understand the nature of human analysts’ cyber SA cognition and decision making. Let this nature inspire innovative designs of SA systems. Break both vertical stovepipes (between compartments) and horizontal stovepipes (between abstraction layers). “Stitched together” awareness enables advanced mission assurance analytics (e.g., asset map, damage, impact, mitigation, recovery). Discover blind spot situation knowledge. Make adversary intent an inherent part of SA analytics.

15. Scientific Principles 15 Cyber security research shows a new trend: moving from qualitative to quantitative science; from data-insufficient science to data-abundant science. The availability of sea of sensed information opens up fascinating opportunities to understand both mission and adversary activity through modeling and analytics. This will require creative mission-aware analysis of heterogeneous data with cross-compartment and cross-abstraction-layer dependencies in the presence of significant uncertainty and untrustworthiness. SA tools should incorporate human cognition and decision making characteristics at the design phase.

16. Why a Multidisciplinary Approach? 16 Several fundamentally important research questions cannot be systematically answered by a single- disciplinary approach. See next slide.

17. Our focus Computer and Information Science of Cyber SA Cognitive Science of Cyber SA Decision Making and Learning Science of Cyber SA Q1: What are the differences between expert analysts and rookies? Q2: What analytics and tools are needed to effectively boost job performance? Q3: How to develop the better tools? 17

18. Technical Approach 18 Draw inspirations from cognitive task analysis, simulations, modeling of analysts’ decision making, and human subject research findings. Use these inspirations to develop a new paradigm of computer-aided cyber SA. Develop new analytics and better tools. Let tools and analysts work in concert. “Green the desert” between the sensor side and the human side. Develop an end-to-end, holistic solution: In contrast, prior work treated the three vertices of the “triangle” as disjoint research areas.

19. The proposed cyber SA framework  The life-cycle side  Shows the to-do SA tasks in each stage of cyber SA  Vision pushes us to “think out-of-the-box” in performing these tasks  The computer-aided cognition side  Research how to build the right cognition models  Research how to build cognition-friendly tools/aids It is a ‘coin’ with two sides: 19

20. Security Analysts Computer network Multi-Sensory Human Computer Interaction Enterprise Model Activity Logs IDS reports Vulnerabilities Cognitive Models & Decision Aids Instance Based Learning Models Simulation Measures of SA & Shared SA Data Conditioning Association & Correlation Automated Reasoning Tools R-CAST Plan-based narratives Graphical models Uncertainty analysis Information Aggregation & Fusion Transaction Graph methods Damage assessment Computer network Real World Test- bed 20 Perception Comprehension Projection

21. Situation Knowledge Abstraction Perspective 21 Mission Workflows Mission Workflows App, Net Services Vulnerability Exploits Alerts OS CPU Liu: integration McNeese & Hall: multi-level cognition and fusion Gonzalez, Cooke Yen, Healey Jajodia, Albanese Subrahmanian Reeves

22. Impact on DoD 22 Significantly enhance mission assurance through: 1. Significantly improving the job performance of CNDSPs 2. Developing cognition-friendly SA tools to effectively improve job performance Situation knowledge integration -- Cross-layer SA analytics Situation knowledge discovery & elicitation Reasoning assistants, decision aids Better interfaces, better workflows

23. 23 Y4 Team Integration Within each theme: Collaboration is pervasive Collaboration is further deepened Joint research tasks Co-authored papers Tool-level integration in progress Between themes: Integration along the functional perspective Integration along the knowledge abstraction perspective E.g., Jajodia & Cooke, Coty & Cooke, Hall & McNeese & Liu, Healey & Hutchinson, Ning & Hutchinson & Jajodia, Yen & Cam & Erbacher & Glodek & Hutchinson & Liu

24. 24 Tech Transfer Deep collaboration with ARL -- ARSCA tool is now being used at ARL to understand the RPs of security analysts -- Adapting ARSCA to directly operate on ARL datasets -- Weekly teleconferences: joint research team DoD STTR that involves a higher fidelity version of CyberCog, DEXTAR, in which we will integrate CAULDRON DoD SBIR 12.3 Phase I OSD12-IA5 project “An Integrated Threat feed Aggregation, Analysis, and Visualization (TAAV) Tool for Cyber Situational Awareness,” funded, led by Intelligent Automation, Inc. (IAI).

25. 25 Tech Transfer (cont’d) The source code for NSDMiner is now released through SourceForge at http://sourceforge.net/projects/nsdminer/. There have been 63 downloads to date.http://sourceforge.net/projects/nsdminer/ Briefings to Deloitte, Lockheed Martin, Raytheon Corporation, MITRE, Computer Sciences Corporation, and MIT Lincoln Laboratory. Briefings to NSA, DTRA, ONR, DHS, and DoDII.

26. Year 5 Plan: Technology Transitions (1) 26 Partner: Contact: Opportunity: Partners: Contacts: Opportunity: Partner: Contact: Opportunity: Partner: Contact: Opportunity: Partner: Contact: Opportunity: AFRL – Human Effectiveness Directorate 711 th Human Performance Wing, Wright-Patterson AFB, OH Benjamin Knott and Vince Mancuso Human performance and measurement of cognition Deloitte, Ernst and Young, KPMG, Price Waterhouse Coopers J.B. O’Kane (Vigilant by Deloitte), Jenna McAuley (EY-ASC) and others Observe practicing analysts, test visualization toolkits and fusion tools, measure human cognition and performance MIT Lincoln Laboratories Cyber Security Information Sciences Division Stephen Rejto and Tony Pensa Conduct human-in-the-loop experiments; evaluate MIT-LL/PSU analyst tools ARL (Tactical Information Analysis) Tim Hanratty Transition knowledge elicitation and visualization toolkits to the demonstration lab at ARL Aberdeen ARL – Adelphi, MD Hasan Cam Applied research in risk and resilience in cyber security

27. Year 5 Plan: Tech Transitions (2) 27 Partner: Contact: Opportunity: Partners: Contacts: Opportunity: Partner: Contact: Opportunity: Partner: Contact: Opportunity: Partner: Contact: Opportunity: ARL (Network division) Bill Glodek, Rob Erbacher, Steve Hutchinson, Hasan Cam, Renee Etoty Tracing and analyzing the reasoning processes of security analysts Sandia Research, Inc. Cooke DoD STTR: A higher fidelity version of CyberCog/DEXTAR/CAULDRON Intelligent Automation, Inc. (Network and Security Division) Jason Li DoD SBIR: Integrated Threat feed Aggregation, Analysis, and Visualization (TAAV) Tool for Cyber Situational Awareness NIST A. Singhal Cloud-wide vulnerability analysis NEC Labs America, Inc. Z. Qian, Z. Li Whole enterprise system-call-level security intelligence

28. Year 5 Plan 28 Each PI has a research plan from their perspectives: see the individual presentations. Per-theme integration exercises will be held. Cross-theme integration exercises will also be held.

29. 29 Q & A Thank you.

30. 30 Our approach: design goals -- Let tools and analysts work in concert -- Fill the space (gap) between the sensor side and the human side -- There needs to be a middle ground -- Sensors and humans do not automatically co-work -- Info floods  acceptable cognition throughput -- Cognition unfriendly analysis  cognition aids -- End-to-end, holistic solution -- The various aspects of cyber SA have been treated as separate problems, in the literature

31. SA is beyond computer security SAComputer Security Gain awareness on both “them” and “us” (Tadda & Salerno). Focus on attacks and attackers. At the end of the day, success is determined by whether the analyst has gained the right situation awareness. Success is determined by whether attacks are blocked, contained, or recovered. Focus on tools.Human centric.

32. 32 Life-cycle side:

33. 33 Computer-aided cognition: bridging the two worlds Logical “World” Mental “World”