Module 1: Why Use Statistics? An Introduction to the ERP Case Study Steve DeGabriele (Massachusetts DEP) For the Workshop: Using Essential Statistics for Effective Program Management Presented at: 2008 Symposium on Innovating for Sustainable Results Chapel Hill, North Carolina January 7, 2008

Overview: Using Essential Statistics for Effective Program Management Module 1 Why use statistics? Why use the Environmental Results Program (ERP) as a case study? Introduction to ERP and its measurement approach Module 2 Key concepts for planning and analyzing a single random sample Orientation to ERP statistical spreadsheet tools Group exercises and discussion Module 3 Key concepts for planning and analyzing comparisons of two random samples Orientation to ERP statistical spreadsheet tools Group exercises and discussion

Why Use Statistics? Statistics are valuable whenever it’s too costly or inefficient to look at everything of interest– whether widgets or dry cleaners Random samples of facilities provide a picture of everyone’s performance, with measurable uncertainty Sample must be “representative” E.g., a sample of volunteer facilities cannot be extrapolated to the population as a whole

Uncomfortable with Statistics? That’s normal, but consider the alternatives Census of all facilities Accurate, but expensive Visiting samples of facilities non-randomly Cheaper, but little sense of how representative the data are

How to Use Statistics As Measurement Tool Primary uses for statistics: Estimate how well a group of facilities is performing at a particular point in time Assess whether performance for a group of facilities has changed over time Assess whether there are differences in performance between two different groups of facilities

What Gets Measured? You can apply statistical analysis to just about any measurable characteristic of a group of facilities Are they in compliance with an important requirement? With all requirements? Are they implementing a key pollution prevention practice? How much hazardous waste are they generating?

How Might This Help You? Statistics can help you assess the need for (and efficacy of)... traditional compliance approaches, innovative environmental performance initiatives, pollution prevention efforts, and voluntary programs. Why are you here? Do you think you might be able to use statistics in your work?

Why Use ERP As a Case Study? ERP’s measurement approach can be used to evaluate group performance... at a single point in time, over time, and across states ERP itself involves elements of... traditional compliance approaches, innovative initiatives, voluntary programs, and pollution prevention

What is ERP? Pioneered by MA DEP in 1990s Printers, dry cleaners, photo processors Integrates proven tools to cost-effectively improve performance in sectors characterized by large numbers of small sources of pollution To date, average initial sector improvements from 5% to 30% Improvement continues in later years Stabilizes at high levels

ERP: Interlocking Tools, Integrated System

ERP: For Large Numbers of Small Pollution Sources Developed by MA in 1990s for printers, dry cleaners and photo processors Hundreds of facilities per sector Significant aggregate environmental footprint Traditional approaches untenable DEP resources shrinking ERP sectors have large numbers of facilities nationally e.g., over 30,000 auto body/paint shops in U. S. So far, six states with auto body ERPs

Current State of ERP Endorsed for "scale up" by EPA Innovation Action Council (IAC) in 2000 Based on MA results, NAPA evaluation and potential to address significant problems Tangible EPA support since then Grants, technical support, tools and resource flexibility States ERP Consortium formed in 2006 Organized as "forum" under ECOS Promotes and supports use of ERP approaches EPA NCEI is a steering committee member

Who's Doing ERP? (18 States, 8 EPA Regions) May 2007

ERP Now Covers 11 Sectors/Groups Small Animal Feedlots (MN) Auto Body Shops (DE, MD*, ME, NY, RI, WA) Dry Cleaners (MA, MI, NH, NV) Oil & Gas Extraction Facilities (LA) Printers (MA, NY, WI) Underground Injection Wells (IL) *FL/MD no longer implementing ERP. MD had one ERP that covered both auto body and auto repair shops

ERP Now Covers 11 Sectors/Groups (Cont.) Auto Salvage Yards (IN, RI) Auto Repair Shops (FL*, MD*) Gas Stations (RI, VA, VT) Photo Processors (MA) Stormwater Dischargers (ME, RI) *FL/MD no longer implementing ERP. MD had one ERP that covered both auto body and auto repair shops

How ERP Works: A Typical ERP Cycle Step 1: Inventory Step 2: Statistical Baseline Inspections Step 3: Compliance Assistance Step 4: Self-Certification Step 7: Informed Decision-Making Step 6: Statistical Post-Certification Inspections Step 5: Targeted Follow-Up Renew Assistance and Certification (As Deemed Necessary)

How ERP Works: A Typical ERP Cycle Identify the myriad small facilities that are sources of pollution, many of which are often unknown to regulators. Step 1: Inventory

How ERP Works: A Typical ERP Cycle Conduct random inspections to accurately measure existing environmental performance and focus outreach on the biggest problems. Step 2: Statistical Baseline Inspections

How ERP Works: A Typical ERP Cycle Work with trade associations to create and provide plain- language, user-friendly assistance that improves compliance and promotes pollution prevention. Step 3: Compliance Assistance

How ERP Works: A Typical ERP Cycle Facilities conduct self-assessments using a detailed checklist closely linked to assistance materials. Responsible officials certify to their facilities’ environmental performance on each item. If necessary, they submit plans to return to compliance. In some cases, certification has been mandatory; in others, voluntary. In all cases, compliance is required. Step 4: Self-Certification

How ERP Works: A Typical ERP Cycle Identify potential problem facilities via certification analysis, and target them for inspections, correspondence or phone calls. Provide assistance and/or initiate enforcement, as needed. Step 5: Targeted Follow-Up

How ERP Works: A Typical ERP Cycle Conduct random inspections to accurately estimate performance changes and verify facility certifications. Step 6: Statistical Post-Certification Inspections

How ERP Works: A Typical ERP Cycle Assess performance data and consider whether to adjust compliance assistance or other strategies directed at the sector or, if sufficient progress has been made over time, target resources elsewhere. Step 7: Informed Decision-Making Renew Assistance and Certification (As Deemed Necessary)

How ERP Measurement Works Performance assessment is largely based upon random, inspector-collected data before and after self-certification Data are representative and objective Recognize that baseline inspections may have an effect on performance It’s part of what you are measuring Critical to follow principles of good data quality, such as: Identify as complete a list of facilities as possible Design a sound inspection checklist (garbage in, garbage out) Train inspectors to follow sampling protocol & collect good and complete data Quality Assurance Project Plans (QAPPs) help many states

Who Gets Inspected? Random samples of all facilities Typically about per round of inspections, sometimes higher Even in voluntary certification programs, you don’t just inspect the volunteers Quality issues with just sampling volunteers: missing the big picture, self-selection bias, and potential to miss spillover effects

What Gets Measured in ERP? States have a lot of flexibility in choosing their measures, but there is convergence Environmental Business Practice Indicators (EBPIs) are basic measurement building blocks EBPIs represent what an agency considers to be the most important environmental aspects for the group Typically multimedia, including both compliance and beyond- compliance issues A multimedia inspection checklist may have well over 100 questions, but most states focus on EBPIs Statistical approaches can be used to infer how well an overall group is performing on EBPIs, based upon just the results from random inspections

EBPIs: Roll-up and Stand-Alone Roll-up indicator Summarizes the results of several questions E.g., a Maryland EBPI tracked the implementation of several emission-reducing painting practices at auto body shops. Maryland observed an increase in use of such painting practices from 40% to 62% of inspected facilities. Stand-alone indicator Linked to one important question on the facility checklist E.g., a Rhode Island auto body EBPI tracked the usage of methylene chloride, a dangerous paint-stripping chemical. Rhode Island observed a decrease in usage from 33% of inspected facilities to 5% of inspected facilities.

Other ERP Measures For simplicity, statistical analysis of EBPIs is the focus of this training course ERP states use statistics to calculate other important measures, too. E.g., Group compliance score: Extent to which the average facility is achieving all relevant compliance-related EBPIs Certification accuracy: Extent to which facility self-certification responses agree with observations of inspectors Change in environmental condition: Extent to which the environment has changed, pollution has decreased, etc., since ERP implementation

Coming Up: A Closer Look at the Two Main ERP Analyses Module 2: Current state of performance Looking at a single random sample Module 3: Difference over time Looking at 2 random samples, before and after certification Difference between states is very similar, but not our focus (for simplicity)

For more information… Steve DeGabriele Director, Business Compliance Division Massachusetts DEP Work: