IEc INDUSTRIAL ECONOMICS, INCORPORATED Attributing Benefits to Voluntary Programs: Practical and Defensible Approaches Cynthia Manson, Principal June 23, 2011

INDUSTRIAL ECONOMICS, INCORPORATED2 Project History EPA ORCR (OSW) faced OMB concerns: Economic benefits of partnership programs Specific ICRs – WasteWise and NPEP Economic efficiency of programs (PART) Identified need to: Respond to demand for robust analysis Noting data limitations of partnership programs Programs already exist, limits analytic options Harmonize discussions of economic analysis and program evaluation Result: Framework for analysis using available data Discussion of limitations of experimental design

INDUSTRIAL ECONOMICS, INCORPORATED3 Economic reasoning for voluntary programs To address market failures: Imperfect information in the marketplace SIGNALING FAILURE Lack of knowledge transfer on green approaches from firm to firm “PUBLIC GOOD” NATURE OF R&D To address unregulated or under- regulated areas, e.g., water conservation, pollution prevention

INDUSTRIAL ECONOMICS, INCORPORATED4 Potential Impacts of EPA Partnership Programs Example Programs: WasteWise, EnergyStar, Natural Gas Star, WaterSense, Green Suppliers Network Provide incentives for participants to share and adopt greener behaviors that, in absence of EPA assistance, would have occurred: Later in time On a temporary or tenuous basis On a smaller scale Not at all

INDUSTRIAL ECONOMICS, INCORPORATED5 Potential Impacts of EPA Partnership Programs “Technical Assistance:” Goal of Information sharing – transfer of R&D, innovation. EPA facilitates transfer of innovations among participants, and to non-participants through web sites and publications. Addresses “public good nature of R&D” Spillover effects DELIBERATE Market signaling: EPA recognition informs consumers about environmental quality, though: Awards and other public recognition; Logos that signal participation and performance; Certification assistance and verification; and Marketing assistance. Addresses “signaling failure”

INDUSTRIAL ECONOMICS, INCORPORATED6 Problem: “Proving” program outcomes Optimal design: randomized control trial (RCT) Strongest approach - addresses causality, attributes program benefits. Requires random assignment of groups to participate and not (drug tests). Random assignment not possible in most EPA contexts, including voluntary programs Alternative to RCT: two-stage approach: Evaluate features of participant group, ensure appropriate selection of control group(s). Approach still requires identifying non-participants. Spillover deliberate – no control group.

INDUSTRIAL ECONOMICS, INCORPORATED7 Proposed Approach: Tiered Assessment with existing data Level 1: Threshold Assessment: ensures and documents that the program design is appropriate for addressing market failure. Level 2: Intervention-Outcome Assessment: verifies that program resources and activities are logically aligned with desired outcomes. Level 3: Quasi-Experimental Design: Quantitative analyses that effectively attribute benefits to the program, while avoiding feasibility issues of experimental design.

INDUSTRIAL ECONOMICS, INCORPORATED8 Level 1: Threshold Assessment: Technical Assistance Threshold evidence for potential technical assistance benefits of a partnership program - innovations are: Non-patentable Applicable broadly to other firms Able to be duplicated by other firms at low cost Able to be duplicated by other firms quickly Applicable to small firms with numerous competitors

INDUSTRIAL ECONOMICS, INCORPORATED9 Level 1: Threshold Assessment: Market signaling Threshold evidence for potential market signaling benefits to a partnership program: Environmental quality characteristics are difficult for the public to observe Environmental quality characteristics are not already addressed by a respected third- party certification of auditing scheme

INDUSTRIAL ECONOMICS, INCORPORATED10 Level 2: Intervention-Outcome Assessment: Thorough inventory of program interventions and outcomes (quantified logic model). Step 1: Information on interventions should include:

INDUSTRIAL ECONOMICS, INCORPORATED11 Level 2: Intervention-Outcome Assessment: Step 2: Information on outcomes should include:

INDUSTRIAL ECONOMICS, INCORPORATED12 Level 2: Intervention-Outcome Assessment: “Logic Model” Example

INDUSTRIAL ECONOMICS, INCORPORATED13 Level 3: Quasi-Experimental Design Examples of quasi-experimental designs: Sub-optimal comparison group: Compare participants and non-participants without statistical correction. Regression discontinuity: Assign participants to a treatment or comparison group on the sole basis of a cutoff score on a pre-program measure. Time series: Measure indicators of study group performance over time, with or without a comparison group. Outcome analysis: Measure changes in outcome variable(s) without accounting for external factors.

INDUSTRIAL ECONOMICS, INCORPORATED14 Illustration of Boyd’s Tiered Approach

IEc INDUSTRIAL ECONOMICS, INCORPORATED