1. Cyber Metrics in the DoD or How Do We Know What We Don’t Know? John S. Bay, Ph.D. Executive Director

2. Things People Have Asked Me How much money should I spend this year on cyber defense technologies? How many attacks has your firewall repelled this month? If I only had a dollar to spend on cyber, where should I spend it? Why is cyber research such a slog? 211/12/14

3. Answers (which did not go over well) How much money have you got? We repelled all of them … except that one you read about in the paper Spend your dollar on upgrades Cyber research is a slog because there is no physics theory underlying it all, liker Maxwells’ Equations or Newton’s Laws 311/12/14

4. But really … it DEPENDS The “threat” factor is common in cybersecurity, but mostly not elsewhere … and it IS true that there is no useful PHYSICS for the problem 411/12/14

5. DoD Taxonomy of Threats 5 From: Defense Science Board, Resilient Military Systems and the Advanced Cyber Threat, January 2013 TierDescription IPractitioners who rely on others to develop the malicious code, delivery mechanisms, and execution strategy (use known exploits). IIPractitioners with a greater depth of experience, with the ability to develop their own tools (from publically known vulnerabilities). IIIPractitioners who focus on the discovery and use of unknown malicious code, are adept at installing user and kernel mode root kits, frequently use data mining tools, target corporate executives a key users (government and industry) for the purpose of stealing personal and corporate data with the expressed purpose of selling the information to other criminal elements. IVCriminal or state actors who are organized, highly technical, proficient, well funded professionals working in teams to discover new vulnerabilities and develop exploits. VState actors who create vulnerabilities through an active program to “influence” commercial products and services during design, development or manufacturing, or with the ability to impact products while in the supply chain to enable exploitation of networks and systems of interest VIStates with the ability to successfully execute full spectrum (cyber capabilities in combination with all of their military and intelligence capabilities) operations to achieve a specific outcome in political, military, economic, etc. domains and apply at scale. 11/12/14

6. And The Corresponding Criticality 611/12/14

7. What Might the COSTS Be? 711/12/14

8. So Then, What to Measure? Qualitative – Capabilities – Missions lost Quantitative – Performance – Cost To achieve Not achieving 811/12/14

9. Capabilities and Maturity 911/12/14

10. Dashboard Approach 1011/12/14

11. “Stoplight Chart” Assessments 1111/12/14 See: SPIDERS JCTD

12. Costs to Us All vulnerabilities are bugs All code has bugs Bugs are expensive Exploits are cheap  the “asymmetry” problem 1211/12/14

13. Mission-Assurance Approach Helps focus attention Requires a “map” o the mission Implies a prioritization on missions (something loses) Requires reconfigurable systems and networks Is not cheap 13 From: DUSD(I&E) Office, HANDBOOK For SELF-ASSESSING SECURITY VULNERABILITIES & RISKS of INDUSTRIAL CONTROL SYSTEMS On DOD INSTALLATIONS, December 2012 11/12/14

14. Just Good Enough (Incremental) Approach How long would our red team take to penetrate the system? – An empirical measure, at best. – Implies a canonical red team 14 prob(first vulnerability is discovered) time Bad code Better code Gamma distribution? 11/12/14

15. The Accountability Approach NIST 800-53 guidelines The “did we do everything we know how to do” approach 15 From: NIST Special Publication 800-53, rev. 4, Security and Privacy Controls for Federal Information Systems and Organizations, April 2013 11/12/14

16. Conclusions: Which is Best? None of them. They service somewhat orthogonal purposes. – But they can provide apples-to-apples comparisons Can they answer the Generals’ questions? – No – … except maybe the one about the firewall – There is CERTAINLY no satisfactory “physics” to guide anybody Cyber Metrics is still an extremely important and high- priority problem for OSD! 1611/12/14