1. Security Metrics That Don’t Suck
Dr Mike Lloyd
CTO

2. Problems in security metrics
Security is the absence of something
Can’t report how often you were NOT on the cover of WSJ

3. How fast is your treadmill?
Many people start with process counting
These measure busyness
Not business
How do you show gains?
Just get busier?

4. An extreme case of measuring busyness
Really bad metrics
An extreme case of measuring busyness
Bad metric → bad behavior → bad outcome

5. On the hamster wheel of pain
Copyright RedSeal Networks, Inc. All rights reserved
Security metrics 1.0
On the hamster wheel of pain
(thanks, Andrew Jaquith!)
Oh look, “stuff” happens …

6. Metrics close the control loop Ops has availability
Management metrics
Metrics close the control loop
Ops has availability
Security needs risk
Focus on outcomes
How easily could a breach occur?
How effective is our spend?
Are we making it harder to break in?
Availability
Operations
Risk
Security

7. Good measurements tell stories Which story depends on who’s on stage
Actors in the play
Good measurements tell stories
Which story depends on who’s on stage
The team member
The CISO
The CFO
Any guesses?

8. The team members and the CISO
Copyright RedSeal Networks, Inc. All rights reserved
Within security
The team members and the CISO
Should we focus on network or endpoints?
What are the top 10 risk contributors?
Are we being effective?

9. The CISO talking to the CFO
Out to the business
The CISO talking to the CFO
Look, we are being effective!
Show reduction in risk
Correlate with expenditures
Ask for more money
Dept of State, Feb 2011

10. About that “asking for money” bit …
The CFO wants ROI
I spend $X, I save $Y
We tell FUD stories
“Look at Sony – want that?”
We’re sick of it, so are they
You have an unexpected ally
Thought about your company’s insurance agent?
Not an actual agent –
real ones are much nicer

11. Insurance brings focus
Data Breach Insurance is available
Pay to transfer risks
They see many breaches
You only see yours
They learn what works
CFO can define “good security” as “that which reduces my insurance premiums”
Measure your posture, negotiate a discount
If we do this right, we could actually measure what we all do for a living!

12. Enough why – time for what

13. Assets you need to protect
Resources
Assets you need to protect
Everyone has some examples
PII, regulatory assets, IP, etc
Some truly “mission critical”
Financial, energy, government, military
Knowledge of vulnerabilities
Bad guys exploit them, so you scan
Counter-measures
It starts with the firewall

14. Simulate attacks before they happen
We want to know our defensive posture
That involves finding the weak points
Attack a model of the network
Measure ease of compromise
Use standards where possible
Copyright RedSeal Networks, Inc. All rights reserved

15. Risk from Network-Based Attacks
Blocking Rule
High Risk
Low Risk
Blocking Rule
Blocking Rule
Pivot Attack
Blocking ACL
Pivot Attack
High Risk
Low Risk
Copyright RedSeal Networks, Inc. All rights reserved

16. Sample attack chain – Before
Internet
DMZ
Main Site
Copyright RedSeal Networks, Inc. All rights reserved

17. Step 1 – Vulnerabilities exposed in DMZ
Attackers can reach these Internet-facing servers
Copyright RedSeal Networks, Inc. All rights reserved

18. Step 2 – Some attack paths sneak in
Just a few pivot attacks are possible
Copyright RedSeal Networks, Inc. All rights reserved

19. An attacker can get in if they find this before you fix it
Step 3 – Attack fans out
An attacker can get in if they find this before you fix it
Copyright RedSeal Networks, Inc. All rights reserved

20. Penetration test results
Sample result:
External attackers can reach red hosts
Then pivot to attack yellow hosts
But no attack combination reached green hosts
Copyright RedSeal Networks, Inc. All rights reserved

21. Turning attacks into measurements

22. Rolling up the scores

23. Dashboards for outbound proof
How easily can attackers get in?
How big is my attack surface?
How much is non-compliant?

24. Dashboards for internal questions
Are investments working?
Where do we need to improve?

25. Defensive posture CAN be measured This drives to better outcomes
Copyright RedSeal Networks, Inc. All rights reserved
Conclusions
Defensive posture CAN be measured
This drives to better outcomes
Measure posture => improved posture
It helps the CFO “get it”
You can sleep better
Demonstrate effectiveness, not busyness

27. Insurance and the search for ALE
As we’ll show, you can measure POSTURE
How easily could someone break in?
True risk tradeoffs require incidence data
Sure, we know a vulnerability is bad …
But how often does it cause losses?
Risk wonks call this Annualized Loss Expectancy

28. Bad breaches are common But for any one company, big ones are rare
Good ALE is hard to find
Bad breaches are common
But for any one company, big ones are rare
Similar to car accidents
Roughly one accident per person per lifetime
How do you get a wide sample of accidents?
Who can test effects of a safety measure?
Insurers can!

29. Another complex arena: chess
Are computers or humans best?
Kasparov: “wrong question”!
How to play the best chess?
Human-computer teams
Humans are good at strategy
Computers don’t get tired
Security faces related problems
Attackers have adopted automation
They will find whatever you miss
Copyright RedSeal Networks, Inc. All rights reserved