Survivability Metrics- A View From the Trenches Partha Pal Rick Schantz, Franklin Webber, Michael Atighetchi DSN METRICS CHALLENGE WORKSHOP 2007 June 27,

Slides:



Advertisements
Similar presentations
Virtual Trunk Protocol
Advertisements

Grounded Theory   Charmaz (2008).
Vulnerability Analysis. Formal verification Formally (mathematically) prove certain characteristics Proves the absence of flaws in a program or design.
Designing Scoring Rubrics. What is a Rubric? Guidelines by which a product is judged Guidelines by which a product is judged Explain the standards for.
DARPA ITS PI Meeting – Honolulu – July 17-21, 2000Slide 1 Aegis Research Corporation Intrusion Tolerance Using Masking, Redundancy and Dispersion DARPA.
Assessment Centre Workshop Budapest How to Succeed at Assessment Centre Jiri Cermak Tomas Vaclavicek Project is funded by.
MODULE 3 1st 2nd 3rd. The Backward Design Learning Objectives What is the purpose of doing an assessment? How to determine what kind of evidences to.
Evaluating Hypotheses Chapter 9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics.
Networking Theory (Part 1). Introduction Overview of the basic concepts of networking Also discusses essential topics of networking theory.
Evaluating Hypotheses Chapter 9 Homework: 1-9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics ~
Specifying a Purpose, Research Questions or Hypothesis
Hands-On Microsoft Windows Server 2003 Networking Chapter 7 Windows Internet Naming Service.
Stephen S. Yau CSE , Fall Security Strategies.
KPIs: Definition and Real Examples
Capability Maturity Model
Introduction to Network Defense
Skybox® Security Solutions for Symantec CCS Comprehensive IT Governance Risk and Access Compliance Management Skybox Security's.
Firewalls CS432. Overview  What are firewalls?  Types of firewalls Packet filtering firewalls Packet filtering firewalls Sateful firewalls Sateful firewalls.
1 Security Risk Analysis of Computer Networks: Techniques and Challenges Anoop Singhal Computer Security Division National Institute of Standards and Technology.
Testing Hypotheses I Lesson 9. Descriptive vs. Inferential Statistics n Descriptive l quantitative descriptions of characteristics n Inferential Statistics.
Visual 3. 1 Lesson 3 Risk Assessment and Risk Mitigation.
1 CREATING A LEARNING ORGANIZATION AND AN ETHICAL ORGANIZATION STRATEGIC MANAGEMENT BUAD 4980.
N By: Md Rezaul Huda Reza n
Strategic planning B.V.L.NARAYANA SPTM. Defining Strategy Strategy is the determinator of the basic long- term goals of an enterprise, and the adoption.
Sociology 3322a. “…the systematic assessment of the operation and/or outcomes of a program or policy, compared to a set of explicit or implicit standards.
Classroom Assessment A Practical Guide for Educators by Craig A
Chapter 5 Section 2 : Storage Networking Technologies and Virtualization.
DSN 2002 June page 1 BBN, UIUC, Boeing, and UM Intrusion Tolerance by Unpredictable Adaptation (ITUA) Franklin Webber BBN Technologies ParthaPal.
1 JUP Conference, Jan 2002 Hosted by Princeton University Quantitative Experimental Results: Automation to Support Time- Critical Replanning Decisions.
Survival by Defense- Enabling Partha Pal, Franklin Webber, Richard Schantz BBN Technologies LLC Proceedings of the Foundations of Intrusion Tolerant Systems(2003)
I-Hack’08 International Hacking Competition “Details”
An Adaptive Intrusion-Tolerant Architecture Alfonso Valdes, Tomas Uribe, Magnus Almgren, Steven Cheung, Yves Deswarte, Bruno Dutertre, Josh Levy, Hassen.
The DPASA Survivable JBI- A High- water Mark in Intrusion Tolerant Systems Partha Pal On Behalf of the Entire DPASA* Team BBN Technologies, Adventium Labs,
First, by sending smaller individual pieces from source to destination, many different conversations can be interleaved on the network. The process.
Institute of Technology Sligo - Dept of Computing Sem 2 Chapter 12 Routing Protocols.
Introduction to Planning
Cisco S3C3 Virtual LANS. Why VLANs? You can define groupings of workstations even if separated by switches and on different LAN segments –They are one.
SRS Architecture Study Partha Pal Franklin Webber.
Slide title In CAPITALS 50 pt Slide subtitle 32 pt Risk-based regression testing in a telecommunication system node Master’s thesis presentation
Chapter Thirteen – Organizational Effectiveness.  Be able to define organizational effectiveness  Understand the issues underpinning measuring organizational.
Differentiation What is meant by differences between learners?
1 Version 3.0 Module 7 Spanning Tree Protocol. 2 Version 3.0 Redundancy Redundancy in a network is needed in case there is loss of connectivity in one.
Software Quality Assurance SOFTWARE DEFECT. Defect Repair Defect Repair is a process of repairing the defective part or replacing it, as needed. For example,
MAVILLE ALASTRE-DIZON Philippine Normal University
By B.Mills Show adapted from The purpose of a network is to share data.
SRS Architecture Study Partha Pal Franklin Webber.
Strategic Planning Session Part II: Outcome Assessment and Program Evaluation Chapter 16: John Clayton Thomas Presented by : David Rudder, Ph.D.
Networking Aspects in the DPASA Survivability Architecture: An Experience Report Michael Atighetchi BBN Technologies.
END OF KEY STAGE TESTS SUMMER TERM 2016.
Virtualized Execution Realizing Network Infrastructures Enhancing Reliability Application Communities PI Meeting Arlington, VA July 10, 2007.
Network Security Principles & Practices By Saadat Malik Cisco Press 2003.
Preparing for the First Hourly. Course Structure Probability  Design Issues and Descriptive Statistics Confidence Intervals and Hypothesis Tests.
1 JUP Conference, Jan 2002 Hosted by Princeton University Experimental Study of Automation to Support Time- Critical Replanning Decisions Kip Johnson,
Automating Cyber- Defense Management By: Zach Archer COSC 316.
QCC General Education Assessment Task Force March 21 and 22, 2016 Faculty Forum on General Education Outcomes.
Sem 2 v2 Chapter 12: Routing. Routers can be configured to use one or more IP routing protocols. Two of these IP routing protocols are RIP and IGRP. After.
Computer Security: Principles and Practice First Edition by William Stallings and Lawrie Brown Lecture slides by Lawrie Brown Chapter 17 – IT Security.
Quality Assurance processes
Chapter 7. Identifying Assets and Activities to Be Protected
Security+ All-In-One Edition Chapter 1 – General Security Concepts
Classroom Assessment A Practical Guide for Educators by Craig A
Outline Introduction Characteristics of intrusion detection systems
AppExchange Security Certification
Significance Tests: The Basics
Capability Maturity Model
Network Architecture By Dr. Shadi Masadeh 1.
Capability Maturity Model
Testing Hypotheses I Lesson 9.
Wireless + TSN = Part of the Picture
Cognitive Support for Intelligent Survivability Management
Presentation transcript:

Survivability Metrics- A View From the Trenches Partha Pal Rick Schantz, Franklin Webber, Michael Atighetchi DSN METRICS CHALLENGE WORKSHOP 2007 June 27, 2007

2 Metrics– What is the Point? (From a system designer’s POV) Evaluating or calibrating emerging technologies or approaches –Is the idea worthwhile? –How good is the realization of the idea? –Does it make any difference? Yardsticks differ from a point solution (e.g., a new IDS or a firewall) to overarching techniques/combination of technologies (e.g., survivability architecture) Evaluating or calibrating a system –Does the system meet the (specified) requirements? –How does the system compare With the undefended version of the same system? With another defended version of the same system? –E.g., two competitive designs With a similar defended system? – E.g., how does a system compare with a known one (say current high- water mark) in terms of defense and survivability characteristics?

3Scoring What to measure so that conclusion about the “point” can be drawn –Binary properties Mission completed? Data revealed? –Time intervals Time taken to achieve some goal Effect of attack on “round trip” –Time based Effect of attack on “frequency” –Count How many attacks/ attack steps? How many failures tolerated? (prevented? Succeeded?) How to measure them –Experiments Red Team Cooperative Red Team Injection –Analytic White boarding Model based Logical arguments –Field Observation Realistic environment Ground truth? Can provide quantitative result Qualitative (subjective) conclusion can be drawn in any of these methods Disproving a Hypothesis

4 The DPASA Red Team Exercise Context Configuration of the defense-enabled system: Nov 2005 Exp3 Exp2 X = /24 Y = /24 Z = /24 X.101 X.102 dpasagw X.99 SeLinux WinXPPro Solaris 8 RedHat ADF NIC Experiment Control Network Win2000 Bump In Wire w/ADF Y.126 Y.125 Y.124 Y.123 Y.122 Y.121 Y.114 SPAN X.126 Q1SM Q1PS Q1COR Q1PSQ Q1DC Q1NIDS Q1AP X.125 X.124 X.123 X.122 X.121 X.114 X.174 Y.174 Y.173 Y.172 Y.171 Y.170 Y.169 Y.162 SPAN Q4SM Q4PS Q4COR Q4PSQ Q4DC Q4NIDS Q4AP X.173 X.172 X.171 X.170 X.169 X.162 X.158 Y.158 Y.157 Y.156 Y.155 Y.154 Y.153 Y.146 SPAN Q3SM Q3PS Q3COR Q3PSQ Q3DC Q3NIDS Q3AP X.157 X.156 X.155 X.154 X.153 X.146 X.142 Y.142 Y.141 Y.140 Y.139 Y.138 Y.137 Y.130 SPAN Q2SM Q2PS Q2COR Q2PSQ Q2DC Q2NIDS Q2AP X.141 X.140 X.139 X.138 X.137 X.130 SPAN Z.146Z.162 Z.130 SPAN Z.114 SPAN AMC CONUS LAN Wing Ops LAN PLANNING LAN ENVIRONMENTAL LAN MAF X.202 Z.202 CombOps X.198 Z.198 AODB X.210 Z.210 TARGET X.211 Z.211 CAF X.212 Z.212 TAP X.213 Z.213 swdistaodbsvrtapdb Hub ChemHaz X.179 Z.179 EDC X.180 Z.180 JEES X.181 Z.181 WxHaz X.178 Z.178 Z.200/29 Z.192/29 Z.208/28 Z.176/28 QUAD 1QUAD 2QUAD 3QUAD 4 Z.128/30Z.144/30Z.112/30Z.160/30 HP 2626 Core Switch VLAN ANIDS X.203 Z.203 ENIDS X.184 Z.184 WNIDS X.196 Z.196 PNIDS X.215 Z.215 Scorebot X.214 Z.214 VLAN 2 Z.113 VLAN 3 Z.129 VLAN 4 Z.145 VLAN 5 Z.161 VLAN 6 Z.177 VLAN 7 Z.209 VLAN 9 Z.193 VLAN 10 Z.201 Cisco 3750 Layer 3 Switch HP 2626 Switch Exp1 X.100 #Bits Mask #Hosts / / / / / / / SPAN Configuration of the defense-enabled system: Exp3 Exp2 X = /24 Y = /24 Z = /24 X.101 X.102 dpasagw X.99 SeLinux WinXPProWinXPPro Solaris 8 RedHat ADF NIC Experiment Control Network Win2000 Bump In Wire w/ADF Y.126 Y.125 Y.124 Y.123 Y.122 Y.121 Y.114 SPAN X.126 Q1SM Q1PS Q1COR Q1PSQ Q1DC Q1NIDS Q1AP X.125 X.124 X.123 X.122 X.121 X.114 X.174 Y.174 Y.173 Y.172 Y.171 Y.170 Y.169 Y.162 SPAN Q4SM Q4PS Q4COR Q4PSQ Q4DC Q4NIDS Q4AP X.173 X.172 X.171 X.170 X.169 X.162 X.158 Y.158 Y.157 Y.156 Y.155 Y.154 Y.153 Y.146 SPAN Q3SM Q3PS Q3COR Q3PSQ Q3DC Q3NIDS Q3AP X.157 X.156 X.155 X.154 X.153 X.146 X.142 Y.142 Y.141 Y.140 Y.139 Y.138 Y.137 Y.130 SPAN Q2SM Q2PS Q2COR Q2PSQ Q2DC Q2NIDS Q2AP X.141 X.140 X.139 X.138 X.137 X.130 SPAN Z.146Z.162 Z.130 SPAN Z.114 SPAN AMC CONUS LAN Wing Ops LAN PLANNING LAN ENVIRONMENTAL LAN MAF X.202 Z.202 MAF X.202 MAF X.202 Z.202 CombOps X.198 CombOps X.198 Z.198 AODB X.210 Z.210 AODB X.210 AODB X.210 Z.210 TARGET X.211 Z.211 TARGET X.211 TARGET X.211 Z.211 CAF X.212 Z.212 CAF X.212 CAF X.212 Z.212 TAP X.213 Z.213 swdistaodbsvrtapdb Hub ChemHaz X.179 Z.179 ChemHaz X.179 ChemHaz X.179 Z.179 EDC X.180 Z.180 JEES X.181 Z.181 WxHaz X.178 Z.178 WxHaz X.178 WxHaz X.178 Z.178 Z.200/29 Z.192/29 Z.208/28 Z.176/28 QUAD 1QUAD 2QUAD 3QUAD 4 Z.128/30Z.144/30Z.112/30Z.160/30 HP 2626 Core Switch VLAN ANIDS X.203 Z.203 ENIDS X.184 Z.184 WNIDS X.196 Z.196 PNIDS X.215 Z.215 Scorebot X.214 Z.214 Scorebot X.214 Scorebot X.214 Z.214 VLAN 2 Z.113 VLAN 3 Z.129 VLAN 4 Z.145 VLAN 5 Z.161 VLAN 6 Z.177 VLAN 7 Z.209 VLAN 9 Z.193 VLAN 10 Z.201 Cisco 3750 Layer 3 Switch HP 2626 Switch Exp1 X.100 #Bits Mask #Hosts / / / / / / / #Bits Mask #Hosts / / / / / / / SPAN PIX VPN Router

5 The DPASA Exercise Had both “technology/approach evaluation” and “survivable system evaluation” flavors –Demonstrate that survivability architecture can be realized –Achieve specific requirements Results must not be tainted by assumptions– Red team was nearly unrestricted Blue needed to focus on many technical details that typically get ignored in a research project– up to date patches, removing unused services, tight policy definitions –Blue team’s task became more complex White team was heavily concerned with measurability –Interpretation of requirements One of the highest priorities for the Red teams was to jeopardize the mission within 12 hrs

6Consequences As a result –The internals of the defended system were not tested –Red team attack focus remained within the external periphery of the system Brute force slow down Zero day exploit to partition COTS VPN Scores like “number of successful server operations” do not reflect the reality Both targeted the COTS routers that connect a local enclave to the IP Network that connects the enclaves

7 Some results Sample Requirement –95% prevention of attacker objectives for 12 hours Interpretation –defended system’s ability to prevent the attacker objectives being met on a per-attack basis. If an attack achieved its stated objective, it was considered successful irrespective of the JBI mission outcome Scoring rules –(p/T*100), where p is the number of attacks that failed to produce the a-priori defined success indicators, and T is total number of attacks Scores –Undefended: 0 out of 17 –Defended Run 1: 6 out of 9 –Defended Run 2: 0 out of 1 Overall impression of the runs –Undefended Run 1: Undefended system was easily taken over –Defended Run 1: Despite delays mission was completed correctly –Defended Run 2: Mission stalled within 1 hr of start Discussion: –Consistent with overall impression (for this requirement)– but what does it say about the quality of defense of the system? –For another requirement the score was not so consistent (undefended system scored higher) –Despite these difficulties, this was a significant attempt in quantifying survivability

8Remedy Cooperative Red team –Red starts the system with pre-positioned attack code that they can control Developed in cooperation with a “traitor” blue team member –Blue cannot act unless they face a symptom They do not know where the attacker is, how many machines are they on or what /how many attacks were going to run –End result of a run: Compressed scenario Defender’s sole objective is to succeed the mission using the defenses integrated in the survivability architecture

9 Some results 75% of such runs resulted in successful mission completion. Required experts to operate the gauges and levers of the defense mechanisms integrated in the architecture

10Conclusion Cooperative Red team exercises stressed the architecture more than the formal red team exercises –Expert participation– next issue for survivability research needs to deal with In line with the notion of self-aware/self-regenerative systems Examples of confidence boosting conclusions (and related metrics) from the system developer/owner’s POV –There is only a small # of attack paths to achieve the high valued targets –The attacker faces multiple # of defenses to achieve any of his objectives More diverse defenses are better –Attacker work factor increased (or decreased) by x % We note that measuring the AWF has typically been a problem –The amount of access and privilege needed by the attacker to succeed has increased (or decreased) Along with an assessment of whether that is reasonable or practical