Measuring What Matters Lisa Young VP Cyber Risk Engineering Axio Global

Data & Information

Terminology – Measure and Metric A measure (or measurement) is the value of a specific characteristic of a given entity (collected data). A metric is the aggregation of one or more measures to create a piece of business intelligence, in context.

Quiz - Measure or Metric? I had 2 eggs for breakfast this morning. It is 48 degrees Fahrenheit in Seattle today. In our organization 3,000 staff have completed the required and updated security awareness training. In our organization 3,000 staff out of 5,000 have completed the required security awareness training since it was updated in January 2018. By March 31, we are on track to ensure 98% of staff have completed security awareness training.

Why do you want to measure?

Getting started Not “What metrics should I use?” but “What do I want to know or learn?” Alternatives: What decisions do I want to inform? What actions do I want to take? What behaviors do I want to change?

Why measure? Speak to decision-makers in their language Demonstrate that the risk management or security program has measureable business value Justify new investments; make improvements Use trends to predict future events Demonstrate that control objectives are (and continue to be) met Answer key questions

Key questions When asked: How secure am I? Am I secure enough? How much risk is acceptable? What does this mean? How secure am I compared to my competition? Am I managing my risks well? What is the business value of being more secure? Of a specific security investment? Do I need to spend more $$ on security or risk management? If so, on what? What are the PR and legal impacts of a data breach?

Measurement objectives -1 Document the purposes for which measurement and analysis are done Specify the kinds of actions that may be taken on the results of data analyses May be identified at the operational unit level or the enterprise level Sources can include Monitoring of risk management process performance Risk conditions Compliance obligations Industry benchmarks Others? Comment re overhead of measurement and analysis; need to be able to demonstrate resilience ROI Dave White has an example about an organization we worked with that gave a project five years to improve resilience

Measurement objectives -2 May include “Reduce the total number of controls under management” “Maintain or improve supplier/customer performance against requirements” “Improve uptime statistics” “Improve risk identification” “Software assets are kept up-to-date based on the criticality of the asset” Once objectives are set, precise and quantifiable measures are established—can be base measure or derived Example of base measure: Number of high-value assets by category Example of derived metric: Percentage of high-value technology assets for which a risk assessment and analysis was conducted in last 12 months

So what? Why do you care? If I had this metric: (*) What decisions would it inform? What actions would I take based on it? What behaviors would it affect? What would improvement look like? What would its value be in comparison to other metrics? (*) informed by Douglas Hubbard, How to Measure Anything, John Wiley & Sons, 2010

Approach State a business objective 4/3/2019 Approach State a business objective Ideally your business objective supports a stated strategic objective Ensure that [business unit, service, product, supply chain, technology, data center] is … available to meet a specified customer or revenue growth objective unavailable for no more than some stated period of time, number of transactions, other units of measure fully compliant with [law, regulation, standard] so as not to incur [z] penalties

Who, what, where, when, why, how? Who is the metric for? Who are the stakeholders? Who collects the measurement data? What is being measured? Where is the data/information stored? When/how frequently are the metrics collected? Why is the metric important (vs. others)? The most meaningful information is conveyed by reporting trends over time vs. point in time metrics. How is the data collected? How is the metric presented? How is the metric used?

To get started Identify sponsors and key stakeholders Define measurement objectives and key questions Determine information that informs these What information do you already have? What information do you need to collect? What is the value of collecting additional information? Define and vet a small number of key metrics data collection analysis procedures number of metrics number of participating business units Collect, analyze, report, refine Leverage an existing measurement program Data visualization and compliance programs

Risk quantification Building a risk quantification method or program is by definition “measuring” something. There are foundational elements that need to be in place for a successful risk quantification program: Business objectives and goals Method and program A set of questions that can be answered with the data; “clean” data Process and workflow; roles and responsibilities Results that are generated from data – minimizes “gaming” and provides context to compare results. Governance and oversight of the method and program We will not get to all of these topics in this workshop. I just wanted you to be thinking about the planning process that would get you to where you want to be.

Cost-effective vs. cost-benefit Cost-benefit – for a given decision, one particular option has both a cost and a benefit. This type of information may not be available on day one when building a measurement program. Cost-effective – desired result or objective achieved by money spent. Generally, this is a better representation of an information security and risk management program. The information provided by the metrics will allow better decision-making. One of the questions to determine an effective security program is to ask: “is behavior changed by measuring X?” If the data is consistently ignored or not considered in decision-making, then the team may want to reconsider what data is being collected and why. (Wong, Security Metrics, pg.23) Data collection and preparation for the analytics to be done are the bulk of the expenses for a measurement program. (Wong, pg.254)

Summary Good metrics are: those that are used often answer important business and stakeholder questions cost little to collect in relation to their value are easily collected do not require extensive manual intervention or manipulation.

Questions Lisa Young Vice President, Cyber Risk Engineering Axio Global LinkedIn: Lyoung@brightmsi.com

GQIM process Objectives Goal Question Indicator Metric Identify business objectives that establish the need for resilience and cybersecurity Goal Develop one or more goals for each objective Question Develop one or more questions that, when answered, help determine the extent to which the goal is met These are the overall steps of the GQIM process that we will discuss in the rest of the workshop. Indicator Identify one or more pieces of information that are required to answer each question Metric Identify one or more metrics that will use selected indicators to answer the question