1 MURI: Computer-aided Human Centric Cyber Situation Awareness Peng Liu Professor & Director, Lions Center Pennsylvania State University ARO Cyber Situation Awareness MURI

Team Peng Liu, Professor and Director, Penn State Center for Cyber-Security, Information Privacy and Trust Massimiliano Albanese, Assistant Professor, GMU Nancy Cooke, Professor and Science Director, Arizona State Cognitive Engineering Research Institute Coty González, Associate Research Professor and Director, CMU Dynamic Decision Making Lab Dave Hall, Professor and Dean, Penn State College of IST Christopher Healey, Professor, NC State 2 Sushil Jajodia, University Professor and Director, George Mason Univ. Center for Secure Information Systems Mike McNeese, Professor and Associate Dean, Penn State College of IST Peng Ning (on leave), Professor, NCSU Douglas Reeves, Professor and Interim Assistant Dean for COE Graduate Programs, NCSU VS Subrahmanian, Professor and past Director, U. of Maryland Institute for Advanced Computer Studies John Yen, University Professor and Director, Intelligent Agents Lab # of post docs: 3 # of graduate students: 16

3 ARO MURI: Computer-aided Human Centric Cyber Situation Awareness PSU, ASU, CMU, GMU, NCSU, UMD Contact: Peng Liu, Tel , Objectives: Improve Cyber SA through: Cyber SA specific cognition models Cognition-friendly tools and analytics that fill the gap between the sensor side and the analyst side of cyber SA Cross-layer situation knowledge integration DoD Benefit: Significantly improved capabilities in gaining cyber SA in face of cyber attacks Significantly improved job performance of analysts Scientific/Technical Approach Take a holistic approach to integrate the “human cognition” aspects and the “cyber tools” aspects of cyber SA Leverage cognition models to develop human cognition-friendly SA techniques, tools, and analytics Accomplishments Year 5: See slide 5 Challenges Understanding the mental processes of analysts Team integration

Security Analysts Computer network Multi-Sensory Human Computer Interaction Enterprise Model Activity Logs IDS reports Vulnerabilities Cognitive Models & Decision Aids Instance Based Learning Models Simulation Measures of SA & Shared SA Data Conditioning Association & Correlation Automated Reasoning Tools R-CAST Plan-based narratives Graphical models Uncertainty analysis Information Aggregation & Fusion Transaction Graph methods Damage assessment Computer network Real World Test- bed 4

5 Year 5 accomplishments Pub: (1 book, 10 journals, 20 conf., 6 chapters) (Y1-Y5 total: 170+) -- 2 PhD thesis -- 7 presentations Tools: -- ARSCA  Shift Transition -- MetaSymploit -- NETS simulator -- DEXTAR -- Patrol  X-Ray -- Switchwall -- NSDMiner -- CyberCog -- PASS  PADUA -- CAULDRON -- etc. Technology transitions: -- See slides later on Research: -- Major achievements made -- See individual presentations Deep collaboration with ARL: ARL security analysts -- 5 researchers at ARL -- 4 joint publications Awards: -- Max Albanese received the 2014 Mason Emerging Researcher/Scholar/Creator Award

What has happened? What is the impact? Why did it happen? What should I do? Security Analysts Computer networks (e.g., GIG) Sensors, probes Cyber Operations for Mission Assurance 6

Cyber Situation Awareness 7 What has happened? What is the impact? Why did it happen? Core Cyber SA Enabler What should I do?

Cyber SA Info Processing Box The Network Attacks Data Sources (feeds) Depicted Situation Ground Truth (estimates) Compare Job Performance 8

Why Research is Needed? CNDSPs*, whose operations are relying on human analysts, face critical challenges: 1. Job performance is unstable 2. Hard to get the big picture: walls between functional domains 3. Better analytics and tools are needed to improve job performance * In the commercial world, similar issues exist.

State of the Art: Big Gap Exists 10 Ability to create problem- solving workflows To see big picture To manage uncertainty To reason albeit incomplete/noisy knowledge To quickly locate needles in haystacks To do strategic planning To predict … Vulnerability scan Event logging Traffic classifying Intrusion detection Alert correlation Signature gen. Taint analysis Back tracking Integrity check Static analysis Bug finding Attack graphs Symbolic execution Sandbox VM monitors … Current tools: BIG GAP Desired cyber SA capabilities:

Scientific Objectives 11 Develop a deep understanding on: 1.Why the job performance between expert and rookie analysts is so different? How to bridge the job performance gap? 2.Why many tools cannot effectively improve job performance? 3.What models, tools and analytics are needed to effectively boost job performance? Develop a new paradigm of cyber SA system design, implementation, and evaluation.

Scientific Barriers 12 A.Massive amounts of sensed info vs. poorly used by analysts B.Silicon-speed info sensing vs. neuron-speed human cognition C.Stovepiped sensing vs. the need for "big picture awareness" D.Knowledge of “us” E.Lack of ground-truth vs. the need for scientifically sound models F.Unknown adversary intent vs. publicly-known vulnerability categories

Potential Scientific Advances 13 Understand the nature of human analysts’ cyber SA cognition and decision making. Let this nature inspire innovative designs of SA systems. Break both vertical stovepipes (between compartments) and horizontal stovepipes (between abstraction layers). “Stitched together” awareness enables advanced mission assurance analytics (e.g., asset map, damage, impact, mitigation, recovery). Discover blind spot situation knowledge. Make adversary intent an inherent part of SA analytics.

Scientific Principles 14 Cybersecurity research shows a new trend: moving from qualitative to quantitative science; from data-insufficient science to data-abundant science. The availability of sea of sensed information opens up fascinating opportunities to understand both mission and adversary activity through modeling and analytics. This will require creative mission-aware analysis of heterogeneous data with cross-compartment and cross-abstraction-layer dependencies in the presence of significant uncertainty and untrustworthiness. SA tools should incorporate human cognition and decision making characteristics at the design phase.

Why a Multidisciplinary Approach? 15 Several fundamentally important research questions cannot be systematically answered by a single- disciplinary approach. See next slide.

Our focus Computer and Information Science of Cyber SA Cognitive Science of Cyber SA Decision Making and Learning Science of Cyber SA Q1: What are the differences between expert analysts and rookies? Q2: What analytics and tools are needed to effectively boost job performance? Q3: How to develop the better tools? 16

Technical Approach 17 Draw inspirations from cognitive task analysis, simulations, modeling of analysts’ decision making, and human subject research findings. Use these inspirations to develop a new paradigm of computer-aided cyber SA Develop new analytics and better tools Let tools and analysts work in concert “Green the desert” between the sensor side and the human side Develop an end-to-end, holistic solution: In contrast, prior work treated the three vertices of the “triangle” as disjoint research areas

The proposed cyber SA framework  The life-cycle side  Shows the SA tasks in each stage of cyber SA  Vision pushes us to “think out-of-the-box” in performing these tasks  The computer-aided cognition side  Build the right cognition models  Build cognition-friendly SA tools It is a ‘coin’ with two sides: 18

Security Analysts Computer network Multi-Sensory Human Computer Interaction Enterprise Model Activity Logs IDS reports Vulnerabilities Cognitive Models & Decision Aids Instance Based Learning Models Simulation Measures of SA & Shared SA Data Conditioning Association & Correlation Automated Reasoning Tools R-CAST Plan-based narratives Graphical models Uncertainty analysis Information Aggregation & Fusion Transaction Graph methods Damage assessment Computer network Real World Test- bed 19 Perception Comprehension Projection

Situation Knowledge Abstraction Perspective 20 Mission Workflows Mission Workflows App, Net Services Vulnerability Exploits Alerts OS CPU Liu: integration McNeese & Hall: multi-level cognition and fusion Gonzalez, Cooke Yen, Healey Jajodia, Albanese Subrahmanian Reeves

Impact on DoD 21 Significantly enhance mission assurance through: 1. Significantly improving the job performance of CNDSPs 2. Developing cognition-friendly SA tools to effectively improve job performance Situation knowledge integration Situation knowledge discovery & elicitation Reasoning assistants, decision aids Better interfaces, better shift transitions

22 Y5 Team Integration Within each theme: Collaboration is pervasive Collaboration is further deepened Joint research tasks Co-authored papers Tool-level integration in progress Between themes: Integration along the functional perspective Integration along the knowledge abstraction perspective E.g., Jajodia & Cooke, Coty & Cooke, Hall & McNeese & Liu, Healey & Hutchinson, Yen & Cam & Erbacher & Glodek & Hutchinson & Liu, Jajodia & Albanese & Cam & Yen & Liu

Technology Transfer (1) 23 Partner: Contact: Focus: Status: ARL Rob Erbacher, Bill Glodek, Steve Hutchinson, Hasan Cam, Renee Etoty, Chris Garneau Collect the cognitive traces of CNDSP analysts -- Over two years -- Over 30 traces collected -- ARSCA tool is being used at ARL -- Weekly teleconferences -- In discussion: directly operate on ARL datasets

Technology Transfer (2) 24 Partner: Contact: Focus: Status: ARL Rob Erbacher, Bill Glodek, Steve Hutchinson Shift transitions -- A user study on shift transition fully designed -- IRB developed and approved -- ARSCA-shift-transition tool developed -- Shipped to ARL site and tested there -- Pilot study is being scheduled

Technology Transfer (3) 25 Partner: Contact: Focus: Status: ARL Hasan Cam Enhance the ARL petri-net model for impact assessment -- feed outputs of CAULDRON and ARSCA into petri-net -- Proposal developed and approved -- Just started (Nov 2014) -- First experiment sketched

Technology Transfer (4) 26 Partner: Contact: Focus: Status: ARL Rob Erbacher, Christopher Garneau (a) Investigate how the current practice of training professional CNDSP security analysts can be enhanced by leveraging ARSCA. (b) A pilot study for investigating the feasibility of using ARSCA-facilitated training procedures for supporting the training of analysts about their analytical reasoning process. -- Proposal developed and approved -- Just started (Nov 2014) -- Weekly teleconferences

Technology Transfer (5) 27 Partner: Contact: Focus: Status: ARL Christopher Garneau, Rob Erbacher Human subject experiments on the cognitive effects of different (visualization) views -- IRB developed and approved -- User study fully designed -- Pilot study being scheduled at Penn State

28 Tech Transfer (6) Phase II STTR: Cooke group has been working with Sushil Jajodia and Max Albanese (George Mason and fellow MURI PIs) on an STTR that involves a higher fidelity version of CyberCog, DEXTAR, in which we will integrate CAULDRON. - Phase II STTR through Sandia Research Corporation - AFRL has shown interest in the test-bed we have been developing on the Phase II STTR Cooke group has been working on SBIR for AFRL with Charles River Associates that involves team sensors for cyber analysts.

Technology Transfer (7) 29 Partner: Contact: Focus: Partners: Contacts: Focus: Partner: Contact: Focus: AFRL – Human Effectiveness Directorate 711 th Human Performance Wing, Wright-Patterson AFB, OH Benjamin Knott and Vince Mancuso Human performance and measurement of cognition Deloitte, Ernst and Young, KPMG, Price Waterhouse Coopers J.B. O’Kane (Vigilant by Deloitte), Jenna McAuley (EY-ASC) and others Observe practicing analysts, test visualization toolkits and fusion tools, measure human cognition and performance MIT Lincoln Laboratories Cyber Security Information Sciences Division Stephen Rejto and Tony Pensa Conduct human-in-the-loop experiments; evaluate MIT-LL/PSU analyst tools

Technology Transfer (8) 30 Partner: Contact: Focus: Status: Partner: Contact: Focus: Status: Partner: Contact: Focus: Status: NIST Anoop Singhal Gain awareness of stealthy info bridges in a cloud -- One research work done -- One NIST technical report produced -- Paper published NEC Labs America, Inc. Z. Qian, Z. Li Discover long-running Idling processes in enterprise systems -- One research work done -- A real enterprise environment (on 24 hosts) -- In-depth measurement study -- Paper submitted IAI, Inc. Jason Li System call level enterprise cyber SA -- A new research work done -- One PhD dissertation

31 Tech Transfer (9) Ethnographic studies/knowledge elicitation with network analysts working in education, military, government, and industry domains. Briefings provided to several companies including: Deloitte, Lockheed Martin, Raytheon Corporation, MITRE, Computer Sciences Corporation, and MIT Lincoln Laboratory. Briefings to NSA, DTRA, ONR, DHS, and DoDII. Neville Stanton, University of Southampton is the developer of EAST modeling and is collaborating with Buchanan and Cooke on this form of modeling applied to cyber.

No-Cost Extension Plan 32 Each PI has a research plan from their perspectives: see the individual presentations Set-aside project 1 with ARL Set-aside project 2 with ARL Team integration exercises will be held

33 Q & A Thank you.