1. The value of the metrics standards within our compliance frameworks
March 2019

2. Not so much who I am but how to contact me
I’ve done stuff, seen things, and made things happen in various places
Joy and bliss
Linked In:
Blog:
Books written: Security for Service Oriented Architecture, 2014 CRC Press

3. The Frameworks 8 “Wholistic” Information Security Frameworks ISO 2700x
CIS CSC
NIST SP-800-XX
NIST CyberSecurity Framework
COBIT
CSA STAR (only applicable to the “cloud”)
PCI DSS (only applicable to the “Payment Card Industry”)
AICPA Trust Services Criteria (Only applicable to “service organizations”)

4. The Frameworks with published standards on metrics
3 “Wholistic” Information Security Frameworks with published metrics
ISO 27004
CIS CSC 7
NIST SP Rev. 1
NIST CyberSecurity Framework
COBIT
CSA STAR (only applicable to the “cloud”)
PCI DSS (only applicable to the “Payment Card Industry”)
AICPA Trust Services Criteria (Only applicable to service organizations)

5. Some Measurements on Metrics Frameworks
ISO 27004
68 Page Standard on Measurements
35 Candidate Measurements
2 kinds of metrics
NIST SP
80 page standard on Performance Measurement
19 Candidate Measurements
3 kinds of metrics
CIS CSC
24 Page guide is a Measurement Companion to the CIS CSC
177 Metrics (one to one relationship between controls and measurements)
Metrics measure completeness of implementation
Presented as six sigma based percent ranges of effectiveness
Except for the controls where control is either 100% or nothing
Assumption: if control is implemented 100% it is effective 100%

6. What does ISO Recommend
Metrics derived from a process with the following roles:
Measurement planner
Defines what is measured and reported to whom
Measurement reviewer
Reviews measurements against requirements
Information owner
Person from whom you must get permission to collect information
Information collector
This may not be a person at all, but a process
Information analyst
This may not be just a person, but likely a person and a process

7. ISO Identifies
2 kinds of measurements
Performance
Effectiveness

8. ISO Process

9. ISO Sources of data Output of various logs and scans
Statistics on training and other human activities
Results of security assessments
Relevant surveys and questionnaires
Incident statistics
Results of internal audits
Results of business continuity / disaster recovery exercises
Management reviews

10. Advice from ISO Make certain of the ease of data collection;
availability of human resources to collect and manage data;
availability of appropriate tools;
number of potentially relevant performance indicators supported by the base measure;
ease of interpretation;
number of users of developed measurement results;
evidence showing the measure’s fitness for purpose or information need; and
costs of collecting, managing, and analyzing the data.

11. Advice from ISO Know the need for the measurement
Identify the most useful formula
Know the implementation evidence needed to be collected
What is the frequency of collection
What is the frequency of reporting
Who collects (automation?)
Who do you report this to?
What format (numbers, pictures, graphs)?

12. Process

13. Candidate Metrics
# of resources allocated to InfoSec compared to original budget
% of Policy reviewed in prior year
Management review meetings scheduled vs performed
# of High and Medium Risks over threshold & number of risks without status updates
# of Audits planned against performed
% of planned Improvement actions on time, on budget, to expectations
Total cost of security incidents
# of improvement actions derived from security incidents
# Corrective actions not implemented compared to those planned

14. More candidate measures
# of employees completing ISMS training
# of employees who take InfoSec Training vs. those who need to take it
Awareness training progress to date
% of employees who pass Awareness campaign test
# of staff who clicked through in phishing training and then did what was asked in the phish compared against number of staff who reported it
Ratio of passwords meeting quality standards tracked over time
Number of passwords cracked in 4 hours vs total number of passwords
% of systems where user access rights are reviewed
# of Physical entry controls systems meeting requirements

15. And yet more candidate measures
Maintenance delay per completed maintenance event
% new systems deployed using change management
# of detected malware actions not blocked
# of systems not running updated Anti-Malware
Availability of systems compared against SLAs
# of unused Firewall rules
% of audit log files reviewed
% of devices configured to policy
% of Penetration Tests per system since last major release

16. Even more candidate measurements
Sum of open CVSS values * number of affected systems to measure effectiveness of vulnerability management
# of security requirements found in 3rd party agreements compared to total number of 3rd party contracts
# security incidents completed within timeframe
Security incident trending
Number of external audits planned against actual

17. What NIST recommends 3 types of measurements: Implementation Measures
How complete is your deployment
Effectiveness/Efficiency Measures
What does success look like
Impact Measures
Positive impact is not always how effective it was
Cost savings
Increase in trust
Negative impact or unintended consequences
Performance impact of scanning for virus
Performance impact of decrypting all traffic for analysis

18. More from NIST Focus is on measurement of controls, not processes
Should be manageable, automated where possible
Reporting on this should
Increase accountability
Improve effectiveness
Demonstrate compliance
Inform decisions
Depends upon
Support from upper management
Procedures in place to supply data
Goals and objectives to compare against dataset

19. Roles and responsibilities
Agent head
Ensures metrics supports strategy
Chief Information Officer
Uses metrics to ensure compliance with requirements
Senior Agency Information Security Officer
Uses metrics to make decisions and run program
Program Manager/Information System Owner
Identifies requirements and provides sources of relevant data
Information System Security Officer
Participates in development of metrics program and collects data

20. Measurements and Goals
Measurements are based upon security performance goals and objectives
Must yield quantifiable information for comparison
Apply formulas for analysis
Track changes using same points of reference

21. Maturity
Over time, the difficulty of measurement should decrease, the ability to automate measurement and analysis increase

23. Metrics Process Phases
Phase 1 Identify stakeholders
Phase 2 Goals and objectives for implementation of NIST SP
These are the minimum security controls
Phase 3 The review of policy, requirements and techniques as to the specifics of how a control are implemented.
These specifics will define what a successful implementation is
Phase 4 Defines source of measurements
Phase 5, 6 & 7
Develop measurements of implementation, effectiveness, impact

24. The process

25. Candidate Metrics Budget % of Organization’s Budget
% of High Vulnerabilities which are Mitigated in X time
% of Remote Access Points used maliciously
% of InfoSec personnel which have received InfoSec training over last year
Average frequency of audit records reviewed for inappropriate activity
% of new systems completed certification before implementation
% of approved and implemented changes identified in latest automated baseline configuration as compared to changes that were not approved
% of systems which have conducted contingency testing

26. More Candidate Metrics
% of users with access to shared accounts
% of incidents reported within required time frame
% of systems whose maintenance was completed on schedule
% of media that passes sanitation procedures before reuse
% of unauthorized physical access
% of employees who sign policy before access is granted
% of employees screened before access is granted
% of vulnerabilities remediated within required time frame
% of contracts with security requirements

27. And yet more candidate metrics
% of mobile computers performing cryptography using FIPS validated cryptographic modules operating in approved modes
% of Operating System Vulnerabilities which have been mitigated

28. CIS CSC

32. Once again, how to contact me
Linked In:
Blog:
Books written: Security for Service Oriented Architecture, 2014 CRC Press