INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental.

Slides:

Advertisements

Similar presentations

Copyright © Allyn & Bacon (2007) Single-Variable, Independent-Groups Designs Graziano and Raulin Research Methods: Chapter 10 This multimedia product and.

Advertisements

Chapter 11: Quasi- Experimental Designs This multimedia product and its contents are protected under copyright law. The following are prohibited by law:

Evaluation Procedures

CHAPTER OVERVIEW The Nonequivalent Control Group Design The Static Group Comparison Single-Subject Design Evaluating Single-Subject Designs.

Designs to Estimate Impacts of MSP Projects with Confidence. Ellen Bobronnikov March 29, 2010.

Quasi-Experimental Design

Quasi Experiments Non-Experimental Research

Research Design and Validity Threats

Wed Oct 29, 2003 Exams after class Mean(SD): 71.6(11.6)% Range: 44-95% Appointment Research day 1 week.

1 Managing Threats to Randomization. Threat (1): Spillovers If people in the control group get treated, randomization is no more perfect Choose the appropriate.

Who are the participants? Creating a Quality Sample 47:269: Research Methods I Dr. Leonard March 22, 2010.

Quasi & Non-Experimental Designs

Types of Group Designs ____________ group design. The experiment compares groups that receive or _______________ the IV (control group) e.g., behavior.

Evaluation Research COMT 502. topics Evaluation research –Process of determining whether intervention has its intended result.

Experimental Research Take some action and observe its effects Take some action and observe its effects Extension of natural science to social science.

Experimental Design The Gold Standard?.

I want to test a wound treatment or educational program but I have no funding or resources, How do I do it? Implementing & evaluating wound research conducted.

+ Controlled User studies HCI /6610 Winter 2013.

Research Methods in Psychology

Research and Evaluation Center Jeffrey A. Butts John Jay College of Criminal Justice City University of New York August 7, 2012 How Researchers Generate.

I want to test a wound treatment or educational program in my clinical setting with patient groups that are convenient or that already exist, How do I.

Overview of MSP Evaluation Rubric Gary Silverstein, Westat MSP Regional Conference San Francisco, February 13-15, 2008.

Selecting a Research Design. Research Design Refers to the outline, plan, or strategy specifying the procedure to be used in answering research questions.

Marius Søberg Norwegian Labour Inspection Authority Lessons from effect assessments.

Copyright © 2008 by Pearson Education, Inc. Upper Saddle River, New Jersey All rights reserved. John W. Creswell Educational Research: Planning,

Chapter 11 Experimental Designs

Power Point Slides by Ronald J. Shope in collaboration with John W. Creswell Chapter 11 Experimental Designs.

COMMON MEASURES PROJECT OVERVIEW Project States: CO, CT, ME, MA, NH, NY, RI, VT, WA An effort to develop and implement common measures for evaluating the.

Copyright ©2008 by Pearson Education, Inc. Pearson Prentice Hall Upper Saddle River, NJ Foundations of Nursing Research, 5e By Rose Marie Nieswiadomy.

Assisting GPRA Report for MSP Xiaodong Zhang, Westat MSP Regional Conference Miami, January 7-9, 2008.

Chapter Four Experimental & Quasi-experimental Designs.

Types of Research and Designs This week and next week… Covering –Research classifications –Variables –Steps in Experimental Research –Validity –Research.

1 Experimental Research Cause + Effect Manipulation Control.

CPUC Workshop on Best Practices & Lessons Learned in Time Variant Pricing TVP Pilot Design and Load Impact M&V Dr. Stephen George Senior Vice President.

Potential and Pitfalls of Experimental Impact Evaluation: Reflections on the design and implementation of an experimental Payments for Environmental Services.

Classifying Designs of MSP Evaluations Lessons Learned and Recommendations Barbara E. Lovitts June 11, 2008.

The World Bank Human Development Network Spanish Impact Evaluation Fund.

Evaluating Impacts of MSP Grants Hilary Rhodes, PhD Ellen Bobronnikov February 22, 2010 Common Issues and Recommendations.

Program Evaluation The use of scientific methods to judge and improve the planning, monitoring, effectiveness, and efficiency of health, nutrition, and.

Chapter 3 should describe what will be done to answer the research question(s), describe how it will be done and justify the research design, and explain.

AmeriCorps Grantee Training Evaluation and Research September 11, 2014.

Impact Evaluation “Randomized Evaluations” Jim Berry Asst. Professor of Economics Cornell University.

What is randomization and how does it solve the causality problem? 2.3.

© 2007 SRI International CPATH Principal Investigators Meeting: Program Evaluation Update March 26, 2010 By the Center for Education Policy Dr. Raymond.

Evaluating Impacts of MSP Grants Ellen Bobronnikov Hilary Rhodes January 11, 2010 Common Issues and Recommendations.

Measuring Impact 1 Non-experimental methods 2 Experiments

1 Module 3 Designs. 2 Family Health Project: Exercise Review Discuss the Family Health Case and these questions. Consider how gender issues influence.

Experimental & Quasi-Experimental Designs Dr. Guerette.

Evaluation Designs Adrienne DiTommaso, MPA, CNCS Office of Research and Evaluation.

The Disability Employment Initiative (DEI): Impact Evaluation Design October 21, 2015 Sung-Woo Cho, Ph.D.

Chapter 11.  The general plan for carrying out a study where the independent variable is changed  Determines the internal validity  Should provide.

Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov February 16, 2011.

Experiments Uniquely suited to identify cause-effect relationships To study effect of one variable (treatment) on another (outcome/dependent variable)

HW 425 Unit 9 Seminar. Agenda  Considerations of Evaluation Design  Experimental vs. Non-Experimental Designs  Benefits and Drawbacks of Design Types.

Randomized Assignment Difference-in-Differences

EXPERIMENTS Lecture 5. Administrative STATA Course Mailing List Info:   No subject  In body.

Outcomes Evaluation A good evaluation is …. –Useful to its audience –practical to implement –conducted ethically –technically accurate.

DRAFT USEPA Office of Compliance Update: 90 CWA Action Plan, State Review Framework, & OECA National Priority Selection Presentation to NACAA Chris Knopes.

IEc INDUSTRIAL ECONOMICS, INCORPORATED Attributing Benefits to Voluntary Programs: Practical and Defensible Approaches Cynthia Manson, Principal June 23,

Evaluation Requirements for MSP and Characteristics of Designs to Estimate Impacts with Confidence Ellen Bobronnikov March 23, 2011.

Experimental Research

Single-Variable, Independent-Groups Designs

Explanation of slide: Logos, to show while the audience arrive.

Designing an Experiment

Pre-post Double Blind Placebo Control Group Design

Impact Evaluation Methods

Quasi-Experimental Design

The Nonexperimental and Quasi-Experimental Strategies

Types of Designs: R: Random Assignment of subjects to groups

Factorial Designs Factorial design: a research design that includes two or more factors (Independent Variables) A two-factor design has two IVs. Example:

Presentation transcript:

INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED Measuring Impact of Compliance Assistance on Auto Body Shops using an Experimental & Quasi- Experimental Evaluation Design Tracy Dyke Redmond Senior Associate June 23, 2011

INDUSTRIAL ECONOMICS, INCORPORATED Typology of Evaluation Designs* Experimental Design Quasi-Experimental Design Non-Experimental Design 2 Increasing Statistical Strength* * Not necessarily overall evaluation strength Pre-test + post-test comparison with statistical matching Regression discontinuity Pre-test + post-test comparison with judgmental matching Pipeline control group Pre-test + post-test comparison with delayed baseline Pre-test + post-test treatment group with post- test only comparison group Post-test only for treatment and comparison group * Not all possible evaluation designs shown Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation,Sage Publications 2006.

INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project for Auto Body Shops Pre-test + post-test comparison with statistical matching Regression discontinuity Pre-test + post-test comparison with judgmental matching Pipeline control group Pre-test + post-test comparison with delayed baseline Pre-test + post-test treatment group with post- test only comparison group Post-test only for treatment and comparison group Treatment and comparison areas selected judgmentally, with subjects randomly selected from within these areas Pro: Flexible Reasonably good estimate of project impact when there are good matching criteria Con: Assumes comparison group similar to treatment group and equally willing to participate Does not assess project implementation Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.

INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project for Auto Body Shops Pre-test + post-test comparison with statistical matching Regression discontinuity Pre-test + post-test comparison with judgmental matching Pipeline control group Pre-test + post-test comparison with delayed baseline Pre-test + post-test treatment group with post- test only comparison group Post-test only for treatment and comparison group Project implemented in phases: subjects in phase 2 serve as control group for subjects in phase 1 Pro: Does not require external control group; design relatively inexpensive and easy to use Con: Assumes phase 1 and 2 groups are similar (which may not be true) Requires that phase 2 group does not have access to phase 1 treatment (spillover problem) Adapted from Bamberger, Rugh, and Margy: RealWorld Evaluation, Sage Publications 2006.

INDUSTRIAL ECONOMICS, INCORPORATED Goal: test impact of EPA compliance assistance (e.g., workshops, webinars, materials) on auto body shop compliance with air and hazardous waste regulations Context: existing hazardous waste regulations, new air regulations coming into effect 2011 (Surface Coating Rule) Requirements: Conduct representative measurement of all regulated entities, not just voluntary participants Do not prevent shops from receiving compliance assistance (or not for very long) Also test phone survey validity (that methodology not covered in this presentation) “Statistically Valid” Pilot Project for Auto Body Shops

INDUSTRIAL ECONOMICS, INCORPORATED Control and Comparison Groups Population = auto body shops located in areas with elevated air toxics risks and subject to the Surface Coating Rule Massachusetts selected as study area because EPA Region 1 planned compliance assistance campaign Treatment and control group in Massachusetts: randomly assigned, considered equivalent Includes auto body shops in eastern MA with elevated risk Excludes communities with pre-existing aggressive assistance/enforcement campaigns Comparison group selected in Tidewater and Piedmont regions of Virginia on the basis of: No expected compliance assistance from EPA or state State regulations related to RCRA and air emissions Number of shops located in areas of elevated-risk

INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project Design MA - A MA - B VA October 2009 – January 2010 March – Early July 2010 Compliance Assistance (CA): Mailings Workshops/ Webinars On-site surveys Followed by on-site CA On-site surveys Summer January 2011 CA: Mailings Webinars CA: Mailings Webinars March – Early July 2010 On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA

INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project Design MA - A MA - B VA October 2009 – January 2010 March – Early July 2010 Compliance Assistance (CA): Mailings Workshops/ Webinars On-site surveys Followed by on-site CA On-site surveys Summer January 2011 CA: Mailings Webinars CA: Mailings Webinars March – Early July 2010 On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA Compare randomly assigned treatment vs. control group in MA

INDUSTRIAL ECONOMICS, INCORPORATED “Statistically Valid” Pilot Project Design MA - A MA - B VA October 2009 – January 2010 March – Early July 2010 Compliance Assistance (CA): Mailings Workshops/ Webinars On-site surveys Followed by on-site CA On-site surveys Summer January 2011 CA: Mailings Webinars CA: Mailings Webinars March – Early July 2010 On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA On-site surveys Followed by on-site CA Compare “difference-in- differences”: Pre-test to Post-test in MA, compared to Pre-test to Post-test in VA

INDUSTRIAL ECONOMICS, INCORPORATED Methodological Notes and Next Steps Among valid shops, response rates between 80 – 85% for both states and both years However, list problems (e.g., shops going out of business) led to need for many “backup shops;” increased cost and effort of project Evidence of shops not on the list, operating “under the radar” in VA Some evidence of spillover problems (e.g., some VA shops may have accessed EPA Region 1 webinars) Currently working to analyze 2011 data and develop comparisons; report expected in late 2011

INDUSTRIAL ECONOMICS, INCORPORATED Thanks to: EPA HQ for developing the vision and supporting the project EPA Region 1 for identify the list of shops, implementing the assistance, and helping conduct site visits EPA Region 3, Virginia, and Massachusetts for participating ERG and Ski Fabyanic for conducting hundreds of site visits Chris Leggett and Michael Crow for helping develop the methodology and analyzing the data The auto body shops, for letting us in the door!

INDUSTRIAL ECONOMICS, INCORPORATED IEc INDUSTRIAL ECONOMICS, INCORPORATED