1. INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental & Quasi- Experimental Evaluation Design Tracy Dyke Redmond Senior Associate June 23, 2011

2. INDUSTRIAL ECONOMICS, INCORPORATED Typology of Evaluation Designs* Experimental Design Quasi-Experimental Design Non-Experimental Design 2 Increasing Statistical Strength* * Not necessarily overall evaluation strength Pre-test + post-test comparison with statistical matching Regression discontinuity Pre-test + post-test comparison with judgmental matching Pipeline control group Pre-test + post-test comparison with delayed baseline Pre-test + post-test treatment group with post- test only comparison group Post-test only for treatment and comparison group * Not all possible evaluation designs shown Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation,Sage Publications 2006.

3. INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project for Auto Body Shops Pre-test + post-test comparison with statistical matching Regression discontinuity Pre-test + post-test comparison with judgmental matching Pipeline control group Pre-test + post-test comparison with delayed baseline Pre-test + post-test treatment group with post- test only comparison group Post-test only for treatment and comparison group Treatment and comparison areas selected judgmentally, with subjects randomly selected from within these areas Pro: Flexible Reasonably good estimate of project impact when there are good matching criteria Con: Assumes comparison group similar to treatment group and equally willing to participate Does not assess project implementation Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.

4. INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project for Auto Body Shops Pre-test + post-test comparison with statistical matching Regression discontinuity Pre-test + post-test comparison with judgmental matching Pipeline control group Pre-test + post-test comparison with delayed baseline Pre-test + post-test treatment group with post- test only comparison group Post-test only for treatment and comparison group Project implemented in phases: subjects in phase 2 serve as control group for subjects in phase 1 Pro: Does not require external control group; design relatively inexpensive and easy to use Con: Assumes phase 1 and 2 groups are similar (which may not be true) Requires that phase 2 group does not have access to phase 1 treatment (spillover problem) Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.

5. INDUSTRIAL ECONOMICS, INCORPORATED Goal: test impact of EPA compliance assistance (e.g., workshops, webinars, materials) on auto body shop compliance with air and hazardous waste regulations Context: existing hazardous waste regulations, new air regulations coming into effect 2011 (Surface Coating Rule) Requirements: Conduct representative measurement of all regulated entities, not just voluntary participants Do not prevent shops from receiving compliance assistance (or not for very long) Also test phone survey validity (that methodology not covered in this presentation) “Statistically Valid” Pilot Project for Auto Body Shops

6. INDUSTRIAL ECONOMICS, INCORPORATED Control and Comparison Groups Population = auto body shops located in areas with elevated air toxics risks and subject to the Surface Coating Rule Massachusetts selected as study area because EPA Region 1 planned compliance assistance campaign Treatment and control group in Massachusetts: randomly assigned, considered equivalent Includes auto body shops in eastern MA with elevated risk Excludes communities with pre-existing aggressive assistance/enforcement campaigns Comparison group selected in Tidewater and Piedmont regions of Virginia on the basis of: No expected compliance assistance from EPA or state State regulations related to RCRA and air emissions Number of shops located in areas of elevated-risk

7. INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project Design MA - A MA - B VA October 2009 – January 2010 March – Early July 2010 Compliance Assistance (CA): Mailings Workshops/ Webinars On-site surveys Followed by on-site CA On-site surveys Summer 2010- January 2011 CA: Mailings Webinars CA: Mailings Webinars March – Early July 2010 On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA

8. INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project Design MA - A MA - B VA October 2009 – January 2010 March – Early July 2010 Compliance Assistance (CA): Mailings Workshops/ Webinars On-site surveys Followed by on-site CA On-site surveys Summer 2010- January 2011 CA: Mailings Webinars CA: Mailings Webinars March – Early July 2010 On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA Compare randomly assigned treatment vs. control group in MA

9. INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project Design MA - A MA - B VA October 2009 – January 2010 March – Early July 2010 Compliance Assistance (CA): Mailings Workshops/ Webinars On-site surveys Followed by on-site CA On-site surveys Summer 2010- January 2011 CA: Mailings Webinars CA: Mailings Webinars March – Early July 2010 On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA Compare “difference-in- differences”: Pre-test to Post-test in MA, compared to Pre-test to Post-test in VA

10. INDUSTRIAL ECONOMICS, INCORPORATED Methodological Notes and Next Steps Among valid shops, response rates between 80 – 85% for both states and both years However, list problems (e.g., shops going out of business) led to need for many “backup shops;” increased cost and effort of project Evidence of shops not on the list, operating “under the radar” in VA Some evidence of spillover problems (e.g., some VA shops may have accessed EPA Region 1 webinars) Currently working to analyze 2011 data and develop comparisons; report expected in late 2011

11. INDUSTRIAL ECONOMICS, INCORPORATED Thanks to: EPA HQ for developing the vision and supporting the project EPA Region 1 for identify the list of shops, implementing the assistance, and helping conduct site visits EPA Region 3, Virginia, and Massachusetts for participating ERG and Ski Fabyanic for conducting hundreds of site visits Chris Leggett and Michael Crow for helping develop the methodology and analyzing the data The auto body shops, for letting us in the door!

12. INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED 617.354.0074