1. Risk Management: Principles of risk, Types of risk and Risk strategies
by Erlan Bakiev, Ph.D.

2. IT Security Metrics
A Practical Approach to Measuring Information Security:
Measuring Security at the System Level

3. Introduction

4. IT Security Metrics Training
Audience: Federal IT security personnel with GISRA reporting responsibilities
Goal: To train Federal IT security personnel how to develop metrics that they can use immediately to assist with GISRA reporting
Duration: 3 hours

5. Objectives After completing this workshop, you will be able to:
Identify why metrics are important for IT security
Identify the relationship among GISRA, NIST SP , and IT Security Metrics
Describe IT Security Metrics
Describe metrics development process
Apply metrics development process by completing a Metrics Form for one of the OMB GISRA reporting requirements for FY02
Identify metrics-related Roles and Responsibilities
Describe how to implement a Metrics Program

6. Metrics Development

7. In this section, you will:
Learn the definition and characteristics of IT Security Metrics
Identify the difference between Performance Goals and IT Security Metrics
Learn the seven-step IT Security Metrics Development Process
Discover the types of information and insights that can be gained from IT Security Metrics
Complete three examples of IT Security Metrics

8. What are IT Security Metrics?
IT Security Metrics are tools that facilitate decision making and accountability through collection, analysis, and reporting of relevant performance data.
Based on IT security performance goals and objectives
Quantifiable
Obtainable/feasible to measure
Repeatable
Provide relevant performance trends over time
Useful in tracking performance and directing resources

9. Why Measure IT Security?
Categories
Requirements
Benefits
Government Information Security Reform Act (GISRA)
Clinger-Cohen Act
Government Paperwork Reduction Act (GPRA)
Regulatory
Satisfy regulatory requirements
Enable investment targeting to identified areas in need
Ensure best value from security
Financial
Measure successes and failures of past and current security investments
Justify future investments
Organizational
Build confidence in leadership
Demonstrate improvement to stakeholders
Play key role in initiating improvement actions based on performance trends
Enable relevant, realistic, appropriate security procedure modification
Improve accountability to stakeholders
Ensure appropriate level of mission support
Determine IT security program effectiveness
Improve customer confidence

10. IT Security Metrics should support IT security goals and objectives
IT Security Performance Goals identify desired results of system security program implementation.
IT Security Performance Objectives enable accomplishment of goals by:
Identifying strategic practices, defined by security policies, procedures, and controls
Directing consistent implementation of policies and procedures across the organization
IT Security Metrics monitor accomplishment of goals and objectives by:
Quantifying the level of implementation of security control objectives and techniques for a system and the effectiveness and efficiency of the controls within the organization
Using analysis of collected IT Security Metrics to determine adequacy of security activities and make appropriate business decisions

11. Exercise: Performance Goal or IT Security Metric?.
Statement
Performance Goal
IT Security Metric
Program Officials understand the risk to systems under their control and determine the acceptable level of risk. .
Percentage of system security plans that are updated annually.
Duties are separated to ensure least privilege and individuals accountability.
Have a separate student and instructor copies when deliver to NIST. The student copy does not contain the checkmarks.
Give the audience a minute to go through items and mark up their copy. Then ask the audience to give answers. If a wrong answer is given, ask the audience to assess whether it is wrong or right.
Percentage of systems with automated virus updating.
Data integrity and validation controls are used to provide assurance that the information has not been altered and the system functions as intended.

12. Metrics development is a seven step process

13. The focus of the metrics program depends on IT security program maturity

14. Stakeholders and Interests
Anyone within an organization is an IT security stakeholder, though some functions have a greater stake than others:
CIO
Program Manager/System Owner
Security Program Manager
Resource Manager
Training/Human Resources Personnel
Each stakeholder needs a set of metrics that provides a view of the organization’s IT security performance within their needs, for a total of no more than metrics per stakeholder
Many IT Security Metrics can be created to measure each aspect of the organization’s IT security. Selecting the most critical elements of the organization’s IT security program during metrics prioritization will make the program manageable and successful

15. IT Security Performance Goals and Objectives
IT security performance goals and objectives are expressed in the form of high level policies and requirements in many laws, regulations, policies, and guidance that describe the dimensions of an effective IT security program:
Clinger Cohen Act
Presidential Decision Directives 63
Government Information Security Reform Act (GISRA)
OMB Circular A-130, Appendix III
Critical Elements within NIST Special Publication
Federal Information Security Compliance Audit Manual (FISCAM)

16. IT Security Policies, Guidance, and Procedures
Some Federal guidance and agency-specific policies and procedures provide more detailed information specific to the agency:
NIST SP ,
Agency-specific policy and guidance
Subordinate questions within NIST Special Publication

17. System Security Program Implementation
System Security implementation includes:
Processes and procedures in place
Existing capabilities
Areas for improvement
Existing metrics
Existing data sources that can be used to derive metrics data
These may be documented in the following sources:
System Security Plans
OMB Plan of Actions and Milestones (POA&M) reports
Latest GAO and IG findings
Tracking of security-related activities
Risk assessments and penetration testing results

18. Metrics can describe three aspects of IT security program operations and management7
Business Impact can be measured through correlation analysis once an organization’s processes are self-regenerating and measurement data gathering is transparent.
Level of Implementation:
Most organizations are new to measuring IT security with performance metrics. They will begin by measuring level of implementation of its security policies and procedures. Instituting a metrics program is the first step to process maturity.
Security Program Effectiveness and Efficiency: As an organization’s process maturity increases and performance data becomes more readily available,
metrics will focus on program efficiency and effectiveness.

19. It is important to record the specifics of each metric for the purposes of data analysis and possible metric reuse
Metric
Describes the overall functionality obtained by collecting the metric
Purpose
Defines the metric by describing the quantitative measurement(s) provided by the metric
Frequency
Implementation Evidence
Specific questions that will need to be answered via survey or through automatic data gathering to be able to calculate the metric
Formula
Frequency
Proposes the periods for collection of data to be used for measuring changes over time. Suggested time periods are based on likely updates occurring in the applicable process
Target
Formula
Describes the calculation to be performed that results in a numeric expression of a metric
Data Source
Lists the location of the data to be used in calculating the metric
Indicators
Provides information about the meaning of the metric and its performance trend. Proposes possible causes of trends, identified through measurement, and points at possible solutions to correct observed shortcomings

20. Metrics can help identify causes of poor performance, including:
Areas
Examples
Resources
Insufficient human, monetary, or other resources can be causing negative performance trends
Lack of appropriate training for the personnel installing, administering, maintaining, or using the systems
Security patches that have been removed during the operating system upgrades
New or upgraded systems that are not configured with all required security settings and patches
Security patches or upgrades that are incompatible with software applications supported by the system
Lack of management awareness and/or commitment to security
Lack of policies and procedures that are required to ensure existence, use, and audit of required security functions
Poor system and security architectures that make systems vulnerable
Inefficient planning processes that influence the metrics (including communication processes necessary to direct organizational actions)
Training
System Upgrades
Configuration Management Practices
Software Compatibility
Awareness and Commitment
Policies and Procedures
Architectures
Inefficient Processes

21. How does NIST SP 800-26 relate to metrics?
Critical Element:
Performance Goal
13.1. Have employees received adequate training to fulfill their security responsibilities?
Subordinate Questions:
Have employees received a copy of the Rules of Behavior?
Is there a mandatory annual refresher training?
Performance Objectives
Are methods employed to make employees aware of security, i.e., posters, booklets?
Have employees received a copy of or have easy access to agency security procedures and policies?
Rules of Behavior are included in training
Employees sign employee agreements stating that they have read and understood rules of behavior
New employee training is conducted
Policies and Procedures discussion included in training
Implementation
Evidence
Annual refresher training is conducted
% of employees who received annual refresher training
% of employees who signed employee agreements
Metrics
% of new employees who underwent security awareness training .

22. Metrics Development Criteria: What is a Good Metric?
 Based on IT security performance goals and objectives: NIST SP Critical Elements and Subordinate Questions are used to derive performance goals and objectives
 Quantifiable: Metrics should yield quantitative rather than qualitative information to increase the objectivity and validity of data
 Obtainable/Feasible to measure: Metrics data should be available or easily collected through interviewing or by accessing data repositories. If a metric requires significant modification of agency processes or implementing a new tool, data collection may not be feasible at this time
 Repeatable: Measurements should be able to be repeated in a standard way at predetermined intervals to identify trends or identify if positive changes have occurred as a result of corrective actions
 Provide relevant performance trends over time: Repeated measurements reveal change in a timely manner
 Useful in tracking performance and directing resources: Metrics should be useful to stakeholders and should yield information that is important in financial decision making

23. Metrics Program Implementation

24. In this section, you will:
Receive an introduction to the IT Security Metrics-related roles and responsibilities
Learn the steps involved in IT Security Metrics program implementation by learning the process and following an example through the process

25. Multiple success factors can influence quality and sophistication of IT Security Metrics
Ensure that IT Security Metrics Program is manageable:
Use no more than metrics at a time, based on current priorities
Phase old metrics out and phase new metrics in when performance targets are reached or when requirements change
Ensure acceptable quality of data:
Data collection methods and data repositories should be standardized
Events must be reported in a standard manner throughout the organization and the results of such reports need to be stored in the data repository

26. Multiple success factors can influence quality and sophistication of IT Security Metrics
Obtain organizational acceptance:
Metrics need to be validated with organization’s stakeholders within headquarters and in the field
Metrics should be vetted through appropriate approval channels
Ensure that metrics are useful and relevant:
Useful data should be collected
Not all data are useful

27. Metrics-related roles and responsibilities are dispersed throughout an organization
Responsibility for
Organizational
Acceptance of
Metrics Program
Responsibility for
Metrics Data
Collection and
Data Accuracy

28. Each organization will implement a metrics program specific to its needs
Tailor to organization and business processes
Identify IT Security Metrics-related stakeholder roles and responsibilities
Lay out required infrastructure changes, such as creation of web-based data collection tools and of new data repositories
Identify required modifications of the current data sources
Define data reporting formats

29. Output from standard security activities can be used to quantify IT security performance
Incident Handling
Testing
Network Management
Audit Logs
Network and System Billing
Configuration Management
Contingency Planning
Training
Certification and Accreditation
IT Security Metrics data collection must be as transparent and non-intrusive as possible.

30. IT Security Metrics Program Implementation Process
Analyze collected data
Conduct gap analysis - Identify gaps between actual and desired performance
Identify reasons for undesired results
Identify areas requiring improvement
Determine range of corrective actions
Select most appropriate corrective actions
Prioritize corrective actions based on overall risk mitigation goals
Develop cost model - Project cost for each corrective action
Perform sensitivity analysis
Develop business case
Prepare budget submission
Identify stakeholders
Determine goals / objectives
Review existing metrics
Develop new metrics
Identify data collection methods and tools
Collect metrics
Track progress
Report as required
Management
Operational
Technical
Budget allocated
Available resources prioritized
Resources assigned

31. Process Implementation Example
Lack of IT security refresher training may be causing weak passwords, identified by a password cracker that is run regularly.
Employees should be required to take annual IT security refresher training as part of their annual review process.
Since annual refresher training has ceased, the number of weak passwords has increased by 50%.
Only 5% of employees receive annual IT security refresher training.
Since the training was re-instituted, the percentage of weak passwords has decreased by 40%
Annual refresher training, an operational control, is instituted.
A budget submission detailing metrics findings related to annual IT security refresher training was submitted, and funding received.

32. Summary Discussed why Metrics are important for IT security
Obtained understanding of the relationship between GISRA, NIST SP , and IT Security Metrics
Described IT Security Metrics
Described the Metrics Development Process
Created metrics to be implemented at a system level through applying metrics development process
Discussed metrics-related Roles and Responsibilities
Described how to implement a Metrics Program