1. Compliance Performance Measure Proposals and Preliminary Trends
Greg Pierce – Chair, Performance Measures Task Force
Compliance and Certification Committee Meeting
March 10-11, 2010

2. Topics PMTF Tasks and Approach Performance Measure Proposals
Preliminary Trends
Outreach Effort
Next Steps

3. PMTF Tasks
Establish an open, consistent and repeatable metric development process to identify and evaluate metrics
Develop compliance improvement measures at the highest levels to provide insight in terms of a set of compliance principles
Coordinate and work with Regional Entities, RMWG, OC, PC and CIPC to ensure agreement
Recommend how to benefit from the compliance improvement measures

4. Approach Adopt RMWG’s open metric development process
Review data source on compliance and other key input factors
Propose a set of performance measures at high levels
Gather data and validate trends
Solicit feedback and other ideas from stakeholders
Prepare a draft report starting April 2010

5. Proposed Performance Measures
Compliance Culture
Timeliness of CVI/CIQ
Average Duration of Mitigation Plan Completion
Violation Risk Index
Average Duration of Audit Report Completion w/ No Violations
Average Duration of Audit Report Completion w/ Potential Violations
Availability/timeliness of 'lessons-learned’ guidance from audits, operating experience, violations and disturbances
Feedback to standards development
The detailed individual metric definitions and specifications are available at

6. Compliance Culture Measure
Metric – Compare Self-Disclosed Violations to the Total Number of Discovered Violations
Self-Disclosed = Self-report + Self-certification
Indicates the level of trust and cooperation within the “NERC Enterprise” as well as the culture of compliance of the stakeholder community

7. Trend By Discovery Method

8. Compliance Culture Trend

9. Timeliness of CVI/CIQ Average Duration – for CVI/CIQ with Open status
Total number of CVIs/CIQs for Open and Closed status
Number of lessons learned and trends issued to industry

10. CVI/CIQ Timeliness Measure
Duration (Days)
CVI
CIQ
Oldest
Average
NERC
644
317
361
242
Region
823
444
126
107

11. Triggers for NERC/Regional CVIs/CIQs

12. Average Duration of Mitigation Plan Completion
Duration – from Date plan received to Mitigation close date
Sum of all durations divided by total number of violations discovered in a particular quarter
Remove zero duration days

13. Average Number of Days for Completing Mitigation Plans

14. Violation Risk Index (VRI)
Violation Risk Index is defined as a risk factor and severity level weighted average based on unmitigated violations
Assess potential consequences of a particular unmitigated violation.
Unmitigated violations of higher risk factor requirements have more weighting value than of lower risk factors.
Determine the change in reliability levels due to Standard requirement violations.
Weighting factors and an example are included in Appendix A.

15. NERC Unmitigated Violations (6/18/07 to 12/31/2009)

16. Violation Risk Trend

17. Average Duration of Audit Report Completion w/ No Violations
Measure audit process efficiency and ability to promptly advise industry of audit results 
These reports do not have potential violations
Starting point: audit ending point
Ending point: report finalized with entity and received by NERC

18. Average Duration of Audit Report Completion w/ No Violations

19. Average Duration of Audit Report Completion w/ Potential Violations
Measure enforcement process efficiency and ability to promptly advise industry of audit results 
These reports have potential violations
Starting point: audit ending point
Ending point: report finalized with entity and received by NERC
Data is not available until second quarter of 2010

20. Timeliness of Lessons-Learned
Establish a common and central platform to share lessons-learned
Timeliness of actionable lesson-learned guidance from operating experience, violations and disturbance events
Lessons-Learned guidance
Recommendations from event analysis
Compliance analysis
Advisories

21. Timeliness of compliance results/feedback to standards
# of SARs (e.g., refine a particular requirement)
# of suggestion and comment submittals
The Standards input form is available at
Direct input to “results-based” standards initiative
Focus on risk-based standards, using first assessment triage sheet

22. Outreach Effort
Chair met with and presented to the RMWG on February 18, 2010
Chair presented to the NERC/Regional Entity Compliance Group at its conference call on February 26, 2010
Chair presented to the OC/PC/CIPC at their joint WebEx conference call on March 9, 2010

23. Next Steps Continue reaching out to seek guidance
Members will meet with OC and PC on March 16-17, 2010
Recommend the proposals in June, desiring CCC approval in September 2010
Comments and suggestions are due April 2, 2010
Send to PMTF Chair, Greg Pierce at and NERC at

24. Question & Answer

25. Appendix A Violation Weighting Factor Table Note:
1. The weighting values are derived by applying similar ratios developed in the Base Penalty Amount Table described in section 4 of the ERO Sanction Guidelines, Appendix 4B to the NERC Rules of Procedure.
2. Reference materials are available in a NERC white paper “Toward Ensuring Reliability: Reliability Performance Metrics”. It can be viewed at:

26. VRI Example