Security Metrics That Don’t Suck Dr Mike Lloyd CTO

Problems in security metrics Security is the absence of something Can’t report how often you were NOT on the cover of WSJ

How fast is your treadmill? Many people start with process counting These measure busyness Not business How do you show gains? Just get busier?

An extreme case of measuring busyness Really bad metrics An extreme case of measuring busyness Bad metric → bad behavior → bad outcome

On the hamster wheel of pain Copyright RedSeal Networks, Inc. All rights reserved Security metrics 1.0 On the hamster wheel of pain (thanks, Andrew Jaquith!) Oh look, “stuff” happens …

Metrics close the control loop Ops has availability Management metrics Metrics close the control loop Ops has availability Security needs risk Focus on outcomes How easily could a breach occur? How effective is our spend? Are we making it harder to break in? Availability Operations Risk Security

Good measurements tell stories Which story depends on who’s on stage Actors in the play Good measurements tell stories Which story depends on who’s on stage The team member The CISO The CFO Any guesses?

The team members and the CISO Copyright RedSeal Networks, Inc. All rights reserved Within security The team members and the CISO Should we focus on network or endpoints? What are the top 10 risk contributors? Are we being effective?

The CISO talking to the CFO Out to the business The CISO talking to the CFO Look, we are being effective! Show reduction in risk Correlate with expenditures Ask for more money Dept of State, Feb 2011

About that “asking for money” bit … The CFO wants ROI I spend $X, I save $Y We tell FUD stories “Look at Sony – want that?” We’re sick of it, so are they You have an unexpected ally Thought about your company’s insurance agent? Not an actual agent – real ones are much nicer

Insurance brings focus Data Breach Insurance is available Pay to transfer risks They see many breaches You only see yours They learn what works CFO can define “good security” as “that which reduces my insurance premiums” Measure your posture, negotiate a discount If we do this right, we could actually measure what we all do for a living!

Enough why – time for what

Assets you need to protect Resources Assets you need to protect Everyone has some examples PII, regulatory assets, IP, etc Some truly “mission critical” Financial, energy, government, military Knowledge of vulnerabilities Bad guys exploit them, so you scan Counter-measures It starts with the firewall

Simulate attacks before they happen We want to know our defensive posture That involves finding the weak points Attack a model of the network Measure ease of compromise Use standards where possible Copyright RedSeal Networks, Inc. All rights reserved

Risk from Network-Based Attacks Blocking Rule High Risk Low Risk Blocking Rule Blocking Rule Pivot Attack Blocking ACL Pivot Attack High Risk Low Risk Copyright RedSeal Networks, Inc. All rights reserved

Sample attack chain – Before Internet DMZ Main Site Copyright RedSeal Networks, Inc. All rights reserved

Step 1 – Vulnerabilities exposed in DMZ Attackers can reach these Internet-facing servers Copyright RedSeal Networks, Inc. All rights reserved

Step 2 – Some attack paths sneak in Just a few pivot attacks are possible Copyright RedSeal Networks, Inc. All rights reserved

An attacker can get in if they find this before you fix it Step 3 – Attack fans out An attacker can get in if they find this before you fix it Copyright RedSeal Networks, Inc. All rights reserved

Penetration test results Sample result: External attackers can reach red hosts Then pivot to attack yellow hosts But no attack combination reached green hosts Copyright RedSeal Networks, Inc. All rights reserved

Turning attacks into measurements

Rolling up the scores

Dashboards for outbound proof How easily can attackers get in? How big is my attack surface? How much is non-compliant?

Dashboards for internal questions Are investments working? Where do we need to improve?

Defensive posture CAN be measured This drives to better outcomes Copyright RedSeal Networks, Inc. All rights reserved Conclusions Defensive posture CAN be measured This drives to better outcomes Measure posture => improved posture It helps the CFO “get it” You can sleep better Demonstrate effectiveness, not busyness

Insurance and the search for ALE As we’ll show, you can measure POSTURE How easily could someone break in? True risk tradeoffs require incidence data Sure, we know a vulnerability is bad … But how often does it cause losses? Risk wonks call this Annualized Loss Expectancy

Bad breaches are common But for any one company, big ones are rare Good ALE is hard to find Bad breaches are common But for any one company, big ones are rare Similar to car accidents Roughly one accident per person per lifetime How do you get a wide sample of accidents? Who can test effects of a safety measure? Insurers can!

Another complex arena: chess Are computers or humans best? Kasparov: “wrong question”! How to play the best chess? Human-computer teams Humans are good at strategy Computers don’t get tired Security faces related problems Attackers have adopted automation They will find whatever you miss Copyright RedSeal Networks, Inc. All rights reserved