Survivability Metrics- A View From the Trenches Partha Pal Rick Schantz, Franklin Webber, Michael Atighetchi DSN METRICS CHALLENGE WORKSHOP 2007 June 27, 2007

2 Metrics– What is the Point? (From a system designer’s POV) Evaluating or calibrating emerging technologies or approaches –Is the idea worthwhile? –How good is the realization of the idea? –Does it make any difference? Yardsticks differ from a point solution (e.g., a new IDS or a firewall) to overarching techniques/combination of technologies (e.g., survivability architecture) Evaluating or calibrating a system –Does the system meet the (specified) requirements? –How does the system compare With the undefended version of the same system? With another defended version of the same system? –E.g., two competitive designs With a similar defended system? – E.g., how does a system compare with a known one (say current high- water mark) in terms of defense and survivability characteristics?

3Scoring What to measure so that conclusion about the “point” can be drawn –Binary properties Mission completed? Data revealed? –Time intervals Time taken to achieve some goal Effect of attack on “round trip” –Time based Effect of attack on “frequency” –Count How many attacks/ attack steps? How many failures tolerated? (prevented? Succeeded?) How to measure them –Experiments Red Team Cooperative Red Team Injection –Analytic White boarding Model based Logical arguments –Field Observation Realistic environment Ground truth? Can provide quantitative result Qualitative (subjective) conclusion can be drawn in any of these methods Disproving a Hypothesis

4 The DPASA Red Team Exercise Context Configuration of the defense-enabled system: Nov 2005 Exp3 Exp2 X = /24 Y = /24 Z = /24 X.101 X.102 dpasagw X.99 SeLinux WinXPPro Solaris 8 RedHat ADF NIC Experiment Control Network Win2000 Bump In Wire w/ADF Y.126 Y.125 Y.124 Y.123 Y.122 Y.121 Y.114 SPAN X.126 Q1SM Q1PS Q1COR Q1PSQ Q1DC Q1NIDS Q1AP X.125 X.124 X.123 X.122 X.121 X.114 X.174 Y.174 Y.173 Y.172 Y.171 Y.170 Y.169 Y.162 SPAN Q4SM Q4PS Q4COR Q4PSQ Q4DC Q4NIDS Q4AP X.173 X.172 X.171 X.170 X.169 X.162 X.158 Y.158 Y.157 Y.156 Y.155 Y.154 Y.153 Y.146 SPAN Q3SM Q3PS Q3COR Q3PSQ Q3DC Q3NIDS Q3AP X.157 X.156 X.155 X.154 X.153 X.146 X.142 Y.142 Y.141 Y.140 Y.139 Y.138 Y.137 Y.130 SPAN Q2SM Q2PS Q2COR Q2PSQ Q2DC Q2NIDS Q2AP X.141 X.140 X.139 X.138 X.137 X.130 SPAN Z.146Z.162 Z.130 SPAN Z.114 SPAN AMC CONUS LAN Wing Ops LAN PLANNING LAN ENVIRONMENTAL LAN MAF X.202 Z.202 CombOps X.198 Z.198 AODB X.210 Z.210 TARGET X.211 Z.211 CAF X.212 Z.212 TAP X.213 Z.213 swdistaodbsvrtapdb Hub ChemHaz X.179 Z.179 EDC X.180 Z.180 JEES X.181 Z.181 WxHaz X.178 Z.178 Z.200/29 Z.192/29 Z.208/28 Z.176/28 QUAD 1QUAD 2QUAD 3QUAD 4 Z.128/30Z.144/30Z.112/30Z.160/30 HP 2626 Core Switch VLAN ANIDS X.203 Z.203 ENIDS X.184 Z.184 WNIDS X.196 Z.196 PNIDS X.215 Z.215 Scorebot X.214 Z.214 VLAN 2 Z.113 VLAN 3 Z.129 VLAN 4 Z.145 VLAN 5 Z.161 VLAN 6 Z.177 VLAN 7 Z.209 VLAN 9 Z.193 VLAN 10 Z.201 Cisco 3750 Layer 3 Switch HP 2626 Switch Exp1 X.100 #Bits Mask #Hosts / / / / / / / SPAN Configuration of the defense-enabled system: Exp3 Exp2 X = /24 Y = /24 Z = /24 X.101 X.102 dpasagw X.99 SeLinux WinXPProWinXPPro Solaris 8 RedHat ADF NIC Experiment Control Network Win2000 Bump In Wire w/ADF Y.126 Y.125 Y.124 Y.123 Y.122 Y.121 Y.114 SPAN X.126 Q1SM Q1PS Q1COR Q1PSQ Q1DC Q1NIDS Q1AP X.125 X.124 X.123 X.122 X.121 X.114 X.174 Y.174 Y.173 Y.172 Y.171 Y.170 Y.169 Y.162 SPAN Q4SM Q4PS Q4COR Q4PSQ Q4DC Q4NIDS Q4AP X.173 X.172 X.171 X.170 X.169 X.162 X.158 Y.158 Y.157 Y.156 Y.155 Y.154 Y.153 Y.146 SPAN Q3SM Q3PS Q3COR Q3PSQ Q3DC Q3NIDS Q3AP X.157 X.156 X.155 X.154 X.153 X.146 X.142 Y.142 Y.141 Y.140 Y.139 Y.138 Y.137 Y.130 SPAN Q2SM Q2PS Q2COR Q2PSQ Q2DC Q2NIDS Q2AP X.141 X.140 X.139 X.138 X.137 X.130 SPAN Z.146Z.162 Z.130 SPAN Z.114 SPAN AMC CONUS LAN Wing Ops LAN PLANNING LAN ENVIRONMENTAL LAN MAF X.202 Z.202 MAF X.202 MAF X.202 Z.202 CombOps X.198 CombOps X.198 Z.198 AODB X.210 Z.210 AODB X.210 AODB X.210 Z.210 TARGET X.211 Z.211 TARGET X.211 TARGET X.211 Z.211 CAF X.212 Z.212 CAF X.212 CAF X.212 Z.212 TAP X.213 Z.213 swdistaodbsvrtapdb Hub ChemHaz X.179 Z.179 ChemHaz X.179 ChemHaz X.179 Z.179 EDC X.180 Z.180 JEES X.181 Z.181 WxHaz X.178 Z.178 WxHaz X.178 WxHaz X.178 Z.178 Z.200/29 Z.192/29 Z.208/28 Z.176/28 QUAD 1QUAD 2QUAD 3QUAD 4 Z.128/30Z.144/30Z.112/30Z.160/30 HP 2626 Core Switch VLAN ANIDS X.203 Z.203 ENIDS X.184 Z.184 WNIDS X.196 Z.196 PNIDS X.215 Z.215 Scorebot X.214 Z.214 Scorebot X.214 Scorebot X.214 Z.214 VLAN 2 Z.113 VLAN 3 Z.129 VLAN 4 Z.145 VLAN 5 Z.161 VLAN 6 Z.177 VLAN 7 Z.209 VLAN 9 Z.193 VLAN 10 Z.201 Cisco 3750 Layer 3 Switch HP 2626 Switch Exp1 X.100 #Bits Mask #Hosts / / / / / / / #Bits Mask #Hosts / / / / / / / SPAN PIX VPN Router

5 The DPASA Exercise Had both “technology/approach evaluation” and “survivable system evaluation” flavors –Demonstrate that survivability architecture can be realized –Achieve specific requirements Results must not be tainted by assumptions– Red team was nearly unrestricted Blue needed to focus on many technical details that typically get ignored in a research project– up to date patches, removing unused services, tight policy definitions –Blue team’s task became more complex White team was heavily concerned with measurability –Interpretation of requirements One of the highest priorities for the Red teams was to jeopardize the mission within 12 hrs

6Consequences As a result –The internals of the defended system were not tested –Red team attack focus remained within the external periphery of the system Brute force slow down Zero day exploit to partition COTS VPN Scores like “number of successful server operations” do not reflect the reality Both targeted the COTS routers that connect a local enclave to the IP Network that connects the enclaves

7 Some results Sample Requirement –95% prevention of attacker objectives for 12 hours Interpretation –defended system’s ability to prevent the attacker objectives being met on a per-attack basis. If an attack achieved its stated objective, it was considered successful irrespective of the JBI mission outcome Scoring rules –(p/T*100), where p is the number of attacks that failed to produce the a-priori defined success indicators, and T is total number of attacks Scores –Undefended: 0 out of 17 –Defended Run 1: 6 out of 9 –Defended Run 2: 0 out of 1 Overall impression of the runs –Undefended Run 1: Undefended system was easily taken over –Defended Run 1: Despite delays mission was completed correctly –Defended Run 2: Mission stalled within 1 hr of start Discussion: –Consistent with overall impression (for this requirement)– but what does it say about the quality of defense of the system? –For another requirement the score was not so consistent (undefended system scored higher) –Despite these difficulties, this was a significant attempt in quantifying survivability

8Remedy Cooperative Red team –Red starts the system with pre-positioned attack code that they can control Developed in cooperation with a “traitor” blue team member –Blue cannot act unless they face a symptom They do not know where the attacker is, how many machines are they on or what /how many attacks were going to run –End result of a run: Compressed scenario Defender’s sole objective is to succeed the mission using the defenses integrated in the survivability architecture

9 Some results 75% of such runs resulted in successful mission completion. Required experts to operate the gauges and levers of the defense mechanisms integrated in the architecture

10Conclusion Cooperative Red team exercises stressed the architecture more than the formal red team exercises –Expert participation– next issue for survivability research needs to deal with In line with the notion of self-aware/self-regenerative systems Examples of confidence boosting conclusions (and related metrics) from the system developer/owner’s POV –There is only a small # of attack paths to achieve the high valued targets –The attacker faces multiple # of defenses to achieve any of his objectives More diverse defenses are better –Attacker work factor increased (or decreased) by x % We note that measuring the AWF has typically been a problem –The amount of access and privilege needed by the attacker to succeed has increased (or decreased) Along with an assessment of whether that is reasonable or practical