Key issues of measuring the effectiveness of compliance assurance Eugene Mazur OECD Environment Directorate

Why measure outcomes? To know that policies and laws are actually implemented! Meeting legal and policy obligations Internal management needs Budget justification Demand for enhanced external accountability The traditional focus on input and output measures is not enough to assess effectiveness! 2

What to measure? Intermediate outcomes – changes in compliance knowledge & behaviour of the regulated community Compliance/noncompliance rates Measures of recidivism/duration of noncompliance Pollution release indicators Indicators of improved environmental management practices and reduced risk Measures of effectiveness of compliance assistance Final outcomes – improvements in environmental conditions as a result of compliance assurance Environmental quality indicators 3

How good are the indicators? Measurability – available at a reasonable cost, regularly updated via a reliable procedure Analytical soundness – well founded in technical and scientific terms –Scope definition, statistical validity, observation periods, link between activities and outcomes Policy relevance – user-friendly, easy to interpret, provides a basis for international comparisons Phase 1 OECD project ( ): analysis of practices in 10 countries 4

Assessment matrix 5 A toolbox – impossible to identify a “best practice” approach or a set of “flawless” indicators.

Key implementation principles Targeting outcome indicators on concrete regulatory priorities (pollutants, sectors) improves analytical soundness but reduces comparability –Develop compliance rates for specific sectors and regulations –Focus pollution release indicators on priority pollutants, use number of pollution incidents –Use only environmental quality indicators that correspond to targeted, problem-oriented efforts Link outcome indicators with time-specific targets Trends analysis in conjunction with input, output indicators greatly increases policy relevance 6

Main issues going forward How to make compliance rates more meaningful? Using composite indices to characterise compliance outcomes? How many compliance indicators to measure (costs and benefits)? Is it feasible to develop a core set of indicators comparable nationally (decentralised systems) and internationally? Phase 2 OECD project ( )

Compliance rates Two main problems with compliance rates: Statistical bias as a result of ever-increasing emphasis on risk-based targeting Binary nature (in or out of compliance) –Differentiate non-compliance based on the potential seriousness of environmental impact 8

Composite indicators Despite the cost and misleading nature of composite indicators (indices), they continue to be developed: –Perceived usefulness as a communication tool –Measure effectiveness on a scale Essentially, three categories of compliance-related indices: –Pollution release indices, for priority pollutants, weighted for their relative harm (NSW, Australia) –Weighted compliance rates to distinguish between procedural and substantive violations (Ontario, Canada) –Risk scores: about more than compliance, but allows a comparable indicator of number of high-risk facilities 9

Number of indicators: costs and benefits Key challenge: how to measure time/cost requirements for data collection of certain types of outcome indicators Prefer intermediate outcome measures with the highest “measurability” (e.g. pollution releases, incidents) Pick those most closely linked to input and output indicators Promising “value for money” projects in the UK and Ireland 10

Comparability Designing comparable indicators not analytically impossible but politically difficult Primary need – in decentralised systems of environmental governance (level playing field across sub-national jurisdictions) Some “candidates”: –Rate of serious non-compliance for 3-4 “marker” requirements (e.g. linked to EU Directives) –Number of high-risk installations (top score band) –Number of serious pollution incidents per number of high-risk installations –Monetary value of complying actions 11