Getting Started: Identifying, Designing, and Using Environmental Compliance & Enforcement Indicators Meredith R. Reeves INECE Secretariat ECENA Workshop on Benchmarking and Setting Environmental Compliance and Enforcement Indicators Szentendre, Hungary 29 March 2007

Overview 1.Introduction 2.Film 3.Identifying Indicators 4.Exercise 5.Designing and Using Indicators 6.Country Example 7.Questions

What is INECE? Global network of environmental compliance and enforcement practitionersGlobal network of environmental compliance and enforcement practitioners –+4,000 from over 120 countries –Government officials, IGOs, and NGO partners Key partners: Netherlands, United States, other national governments, World Bank, OECD, European Commission, UNEP, regional enforcement networksKey partners: Netherlands, United States, other national governments, World Bank, OECD, European Commission, UNEP, regional enforcement networks Secretariat based in Washington, D.C.Secretariat based in Washington, D.C. Executive Planning Committee guides implementation of mission and goalsExecutive Planning Committee guides implementation of mission and goals

INECE Goals Raise Awareness to Environmental Compliance and Enforcement at each step in the Regulatory CycleRaise Awareness to Environmental Compliance and Enforcement at each step in the Regulatory Cycle –International Conferences –Issue Analysis and Publications Build Capacity for Environmental Compliance and EnforcementBuild Capacity for Environmental Compliance and Enforcement –Annotated Principles of Environmental Enforcement –Training Course on ECE & Legal Aspects of Water Governance Strengthen Networks for Enforcement CooperationStrengthen Networks for Enforcement Cooperation –Hazardous Wastes, Water Governance, Environmental Crime, Judiciary, Illegal Logging, Ozone Depleting Substances

Environmental Compliance and Enforcement Indicators Flagship INECE projectFlagship INECE project Initiated in 2002Initiated in 2002 Main Activities & Outputs:Main Activities & Outputs: –Expert Working Group –Performance Measurement Guidance for Compliance and Enforcement Practitioners –Capacity Building Program –Executive-level Brochure –Assist in the formation of pilot projects in various countries

What are Environmental Compliance & Enforcement Indicators? IndicatorsIndicators –From Latin verb indicare, to make known or point out –pieces of information that provide evidence on matters of broader concern Environmental compliance and enforcement indicatorsEnvironmental compliance and enforcement indicators –Inform users about the performance of and the results achieved by ECE programs

What Types of Indicators Are Needed for ECE Programs? Input: personnel, funds, other resources that contribute to an activityInput: personnel, funds, other resources that contribute to an activity Output: quantitative or qualitative measures of activities, work products, or actionsOutput: quantitative or qualitative measures of activities, work products, or actions –Example: number of enforcement actions taken in 2005

What Types of Indicators Are Needed for ECE Programs? Intermediate Outcome: changes in knowledge, behavior, or conditions that result from program activities and are needed to achieve the end outcomeIntermediate Outcome: changes in knowledge, behavior, or conditions that result from program activities and are needed to achieve the end outcome –Example: pounds of pollution reduced End Outcome: the ultimate results a program is designed to achieveEnd Outcome: the ultimate results a program is designed to achieve –Example: improvement in air quality

Benefits of Indicators to Benchmarking Quantitative and qualitative performance measurement indicators are key components of the benchmarking methodology 1.Assess current program operation parameters 2.Compare program activities with external partner 3.Measure progress towards recommendations 4.Identify and help correct performance issues 5.Inform resource allocation decisions 6.Communicate progress to decision-makers and the public

What Purposes Do Indicators Serve? Purpose: Evaluate Question: How far have we progressed towards the recommended changes? Adapted from Behn (2003)

What Purposes Do Indicators Serve? Purpose: Budget Question: How should we spend public funds?

What Purposes Do Indicators Serve? Purpose: Motivate Question: How can we use information from an external program to motivate improvements in performance?

What Purposes Do Indicators Serve? Purpose: Promote Question: How can we demonstrate our agency is doing a good job?

What Purposes Do Indicators Serve? Purpose: Recognize Question: What accomplishments are worthy of recognition?

What Purposes Do Indicators Serve? Purpose: Control Question: How can we ensure that the activities of our program staff are leading to improved performance?

What Purposes Do Indicators Serve? Purpose: Learn Question: What is working or not working? Why or why not?

What Purposes Do Indicators Serve? Purpose: Improve Question: What should we do differently to improve performance?

INECE Film on Environmental Compliance and Enforcement Indicators what gets measured, gets managed

Best Practices Determine scope Apply logic model Develop guiding principles Select criteria for evaluating indicators Develop common definitions for key terms Inventory existing data sources Look beyond existing data Select appropriate combination of indicators Stage 1 Identifying Potential Indicators Consult with stakeholders and staff Best Practices Use internal teams to determine how to design and test Conduct pilot projects Develop in phases Consult with experts Monitor design and testing Create and distribute development plan Ensure timely and accurate reporting Stage 2 Designing Indicators Stage 3 Using Indicators Monitor performance with regular reports Assess and adapt indicators Analyze performance of organizational units Review effectiveness of specific programs Report to external audiences Analyze behind the numbers Three-Stage Model

Stage 1: Identifying Indicators

Identifying ECE Indicators Define scope of indicators neededDefine scope of indicators needed –Are you conducting process benchmarking or performance benchmarking? –Measuring national, regional or local program? –Measuring whole program or specific initiative or problem?

Identifying ECE Indicators Conduct meetings with external stakeholders and internal staff to solicit:Conduct meetings with external stakeholders and internal staff to solicit: –Potential performance measures –Criteria for evaluating potential measures –Principles to guide the overall effort to develop and use indicators

Establish definitions for types of indicatorsEstablish definitions for types of indicators –Define inputs, outputs, outcomes Create a “map” of activities and results, perhaps using logic modelCreate a “map” of activities and results, perhaps using logic model –Identify full range of results of program activities, with particular attention to intermediate outcomes Identifying ECE Indicators

Sample Logic Model InputsresourcesOutputsactivities Intermediate Outcomes behavior change Final Outcomes environmental impact Personnel Funds for salaries, contracts, etc. Inspections conducted Enforcement actions taken Fines assessed and collected Greater understanding of how to comply Improved environmental management practices Increased compliance Reduced pollution emissions Improved ambient water quality Reduced contaminant burden in wildlife species

Identifying ECE Indicators Inventory existing data sourcesInventory existing data sources –Determine whether existing data will support potential indicators –Limitations of data sources should not limit consideration of potential indicators Select appropriate combinations of indicators for further developmentSelect appropriate combinations of indicators for further development –Use Selection Criteria –Make important audiences aware of which indicators have been selected for development

Sample Criteria for Evaluating Performance Indicators Relevant: to goals, objectives, and prioritiesRelevant: to goals, objectives, and priorities Transparent: promotes understanding of programTransparent: promotes understanding of program Credible: based on data that is complete and accurateCredible: based on data that is complete and accurate Functional: encourages constructive behaviorFunctional: encourages constructive behavior Feasible: value to program outweighs costFeasible: value to program outweighs cost Comprehensive: covers important operational aspectsComprehensive: covers important operational aspects

Indicator Type Identification Exercise

Stage 2: Designing Indicators

Designing ECE Indicators Select indicators for design and implementationSelect indicators for design and implementation Establish internal work teams to:Establish internal work teams to: –Define measures in more detail –Review relevant data in existing data systems –Develop new information collection and reporting processes –Establish schedule for testing and developing indicators

Designing ECE Indicators Conduct pilot projects to test indicators and correct problemsConduct pilot projects to test indicators and correct problems Use consultants as needed to resolve technical and methodological issuesUse consultants as needed to resolve technical and methodological issues Execute implementation in phasesExecute implementation in phases

Designing ECE Indicators Monitor the design and testingMonitor the design and testing Create and distribute a development planCreate and distribute a development plan Ensure timely and accurate reportingEnsure timely and accurate reporting

Stage 3: Using Indicators

Using Indicators Monitor performance through regular reportsMonitor performance through regular reports Review performance of organizational unitsReview performance of organizational units Evaluate effectiveness of specific programsEvaluate effectiveness of specific programs

Suggested Practices for Using ECE Indicators Analyze behind the numbersAnalyze behind the numbers Report to external audiencesReport to external audiences Assess and adapt indicatorsAssess and adapt indicators Assess and adapt enforcement programAssess and adapt enforcement program

Possible Challenges in Using Indicators Duration of implementationDuration of implementation Lack of interpretive skillsLack of interpretive skills Misuse by external audiencesMisuse by external audiences Inherent limitations of indicatorsInherent limitations of indicators

Country Example US Environmental Protection Agency Office of Enforcement and Compliance Assurance (OECA)

Program Foundations Government Performance and Results Act (GPRA) of 1993Government Performance and Results Act (GPRA) of 1993 –Requires Annual Performance Report on results and outcomes achieved through the activities of major federal programs Executive level focus on:Executive level focus on: –outcome measures to monitor program performance –performance-based management in ECE Programs

Program Goals Goal 1: Report on results achieved to external audiences the public,the public, the U.S. Congress, andthe U.S. Congress, and the U.S. Office of Management and Budget (OMB)the U.S. Office of Management and Budget (OMB)

Examples of EPA Outcome Indicators Pounds of pollutants reduced by enforcement actionsPounds of pollutants reduced by enforcement actions Pounds of soil treated or removedPounds of soil treated or removed Gallons of groundwater treatedGallons of groundwater treated Number of companies/facilities self- disclosing and correcting violations under EPA Audit PoliciesNumber of companies/facilities self- disclosing and correcting violations under EPA Audit Policies

Criminal Enforcement Program FY 2001 – FY 2005 FY 2005 Data Source: Criminal Docket, October 27, Disclaimer: Due to enhanced data quality reviews, minor corrections may have been made to previously reported data. As such, this FY 2005 End-of-Year Report contains updated enforcement and compliance data for prior years.

Sentencing Results FY 2001 – FY 2005 FY 2005 Data Source: Criminal Docket, October 27, Disclaimer: Due to enhanced data quality reviews, minor corrections may have been made to previously reported data. As such, this FY 2005 End-of-Year Report contains updated enforcement and compliance data for prior years.

FY 2005 Data Sources: Integrated Compliance Information System (ICIS), October 27, 2005 and on-line usage report. Disclaimer: Due to enhanced data quality reviews, minor corrections may have been made to previously reported data. As such, this FY 2005 End-of-Year Report contains updated enforcement and compliance data for prior years. Entities Reached with Compliance Assistance

FY 2005 Highlights In FY 2005, actions from the enforcement and compliance program produced significant results including… Enforcement actions projected to reduce pollution by 1.1 billion poundsEnforcement actions projected to reduce pollution by 1.1 billion pounds Defendants will spend a record $10 billion to reduce pollution and achieve complianceDefendants will spend a record $10 billion to reduce pollution and achieve compliance Defendants will pay $100 million in criminal fines and restitution and were sentenced to 186 years in jailDefendants will pay $100 million in criminal fines and restitution and were sentenced to 186 years in jail Results like this reflect a strong and active enforcement program, effective targeting, and a focus on results-based performance.

Program Goals Goal 2: Monitoring, Analyzing, and Improving Performance Monthly Management ReportsMonthly Management Reports year-to-date status of activities reported to senior managersyear-to-date status of activities reported to senior managers Regional Data ProfilesRegional Data Profiles twice yearly status on performance in the current fiscal year, three-year trends on key outputs and outcomes,twice yearly status on performance in the current fiscal year, three-year trends on key outputs and outcomes, comparisons between regionscomparisons between regions Watch List for Significant NoncomplianceWatch List for Significant Noncompliance List of facilities where significant noncompliance has been identified but has not been addressed after a prolonged periodList of facilities where significant noncompliance has been identified but has not been addressed after a prolonged period

Questions for Consideration What do you anticipate being your biggest institutional hurdle?What do you anticipate being your biggest institutional hurdle? Who should be participating on your team to design the indicators? What groups should be represented?Who should be participating on your team to design the indicators? What groups should be represented?

Contact Information INECE Secretariat 2300 Wisconsin Ave, NW Suite 300B Washington, DC Phone: Web: