Cost-Effectiveness Testing & Non-Energy Impacts in Program Evaluation Jackie Berger Home Performance Conference April 25, 2018

NON-ENERGY IMPACT LITERATURE 2

Types of Residential NEIs Benefit Level Societal Household Benefits To General Population Ratepayer Participant Economic  Environmental Health & Safety Affordability Collection Costs System Reliability Indoor Air Quality Noise Water Usage Maintenance 3

Benefits to Ratepayers and Participants Household level benefits may result in benefits to society Program participant health improvements May lead to reductions in Medicaid or Medicare costs Benefits accrue to taxpayers Household level benefits may result in benefits to ratepayers Program participant affordability improvements May lead to reductions in the cost of bill payment subsidy Benefits accrue to ratepayers 4

Challenges in the Literature Not applicable to current study Utilizes past estimates Out of date Robustness of approach Sample size Statistical significance Estimate quality not assessed Methodology Assumptions Limitations Lack of documentation 5

Common Approach To NEI Estimation Review past studies Select particular benefits for inclusion Take an average of past studies or subset of reviewed studies 6

Important Questions Methodology Design Assumptions Sample size Statistical significance Robustness Causality Response rate Impact of attrition Comparison group quality Factors Population served Program design Program delivery Weather Fuels Baseline Other services available 7

MEASURING NEIs 8

Measurement Approaches: H&S, Comfort, Home Condition Home Measurements Direct on-site assessments H&S: Mold, moisture, asbestos, CO, radon Comfort: Temperature, humidity Health Records Health conditions Medical utilization Police and Fire Department Data Incidence of fires & accidents Not usually available at unit level Occupant Reports Home conditions, comfort, health, safety Prior to and following treatment 9

National WAP Evaluation Measurement Framework Design: Pre and post-treatment surveys Measured housing unit and health status for participants and comparison group Prior to weatherization At least one year after weatherization Treatment Group Households scheduled for a WAP audit. Comparison Group Treated by WAP one year prior to survey. Baseline Survey Conducted summer 2011 Collected client self-reports of housing unit conditions and health status Follow-Up Survey Conducted summer 2013 Used same survey questions Analysis Restricted to homes that were weatherized and still occupying weatherized home in 2013 Compared changes for Treatment Group to changes for Comparison Group 10

National WAP Evaluation Measurement Framework Quasi-Experimental Analysis Design   Pre Post Change Measured Treatment Before Audit 12-18 Months After Services Pre-Post Program Impact + Other Factors Comparison After Services One Year Later Post-Post Other Factors Treatment - Comparison [Treatment Pre – Treatment Post] – [Comparison Post1 – Comparison Post2] Program Impact 11

National WAP Evaluation Design Limitations Earlier participants instead of later participants Less room for impact from external factors Potential understatement of impact of exogenous factors Comparison Group Differential attrition Very cold climate is over-represented Hot climate is under-represented Longitudinal Response Rate Income (continuing recovery from recession) LIHEAP receipt (referral to WAP following LIHEAP application) Omitted Variables >9% changed asthma response Additional call for clarification 19% different respondent and could not verify, 28% not reached Measurement Issues 12

National WAP Evaluation Affordability Impacts   Treatment Group Comparison Group Net Change Pre Post Change Very hard/hard to pay energy bills 76% 49% -26%*** 58% 52% -6%** -20%*** Could not pay energy bills at least every few months 17% 14% -3% 13% 12% -1% -2% Used short-term, high interest loan to pay for energy bills 15% 9% Could not buy food at least every few months to pay for energy bills 10% -4%** 8% Could not fill prescription at least every few months to pay for energy bills -6%*** 6% -3%** 13

National WAP Evaluation Bill Payment Impacts   Treatment Group Comparison Group Net Change Pre Post Change Received disconnect notice almost every month 16% 12% -4%** 13% 11% -2% -1% Electricity or gas disconnected 3% 2% 0% Fuel ran out 9% 5% -4%*** 4% -3%* 14

National WAP Evaluation Home Condition Impacts   Treatment Group Comparison Group Net Change Pre Post Change Extremely/very infested with cockroaches, spiders, other insects 5% 2% -3%** 0% Somewhat infested with cockroaches, spiders, other insects 19% 12% -7%*** 13% 15% 3% -10%*** Extremely/very infested with rats or mice -2%*** 1% Somewhat infested with rats or mice 8% 6% -2% 15

National WAP Evaluation Home Condition Impacts   Treatment Group Comparison Group Net Change Pre Post Change Mildew odor or musty smell 29% 21% -8%*** 15% 16% 1% -10%*** Mold 24% 19% -5%** 17% -1% -4% Always or often observed standing water 5% 4% 3% 0% Sometimes observed standing water 9% -6%*** 7% 16

National WAP Evaluation Home Condition Impacts   Treatment Group Comparison Group Net Change Pre Post Change A great deal of noise 28% 17% -12%*** 12% 0% Drafty all the time 2% -10%*** 4% 3% -1% -9%*** Indoor temperature is comfortable in the winter 58% 82% 23%*** 79% 83% 4%* 20%*** Indoor temperature is comfortable in the summer 57% 71% 13%*** 72% 74% 12%*** Never have unsafe or unhealthy indoor temperature 81% 93% 91% 17

National WAP Evaluation Health Impacts  Respondent Health Treatment Group Comparison Group Net Change Pre Post Change Has ever had asthma 19% 21% 2%*** 0% Allergies 28% 31% 3% 7%*** -4% The flu 18% -3% 16% -1% -2% Persistent cold symptoms more than 14 days 20% 14% -6%*** 17% Sinusitis 37% 33% -4%* 34% 18

National WAP Evaluation Safety Impacts   Treatment Group Comparison Group Net Change Pre Post Change Has working smoke detectors 90% 97% 7%*** 96% -1% 8%*** Has working CO monitors 44% 80% 36%*** 76% 73% -4%* 40%*** Fire dept. called to put out fire 1% 0% -1%** Food poisoned in home and went to see a medical professional -1%* Poisoned by CO, and went to see a medical professional Lead poisoned and went to see a medical professional Burned from water coming out of a faucet or showerhead in home 19

Ideas for Improved Measurement Difficult to track receipt of services using client response At state level could use program records Program Records Assess measures installed in each home Compare outcomes for households that received particular measures Weatherization Services LIHEAP participation Better assessment of affordability issues Additional Questions Need to keep testing health status questions Improved Questions 20

Challenge Monetizing the Impact 40% increase in homes with working CO detectors 20% increase in homes with comfortable winter temperature 10% reduction in homes with mildew odor or musty smell Examples of Measured Impacts Develop estimates of potential harmful effects and the cost of those Ask participants to value compared to the energy savings How much is that worth $? 21

New Jersey Natural Gas SAVEGREEN PROGRAMS 22

SAVEGREEN Programs Goals Increase energy efficiency opportunities for customers Promote and enhance the use of the NJCEP offerings Raise awareness of the whole house approach to energy efficiency Increase customer awareness of energy efficient appliances and weatherization measures Increase NJ employment in energy efficiency and conservation 23

SAVEGREEN Programs Overview NJ Clean Energy Program SAVEGREEN Project Jan-June July-Dec Furnace/Boiler Enhanced Rebate Furnace $400 $250 $900 $500 Boiler $300 NJNG Audit Required Furnace and Water Heater $6500 OBRP Up to $6,500 OBRP 0% interest over 5 years HPwES Tier Savings Rebate Up to $10,000 OBRP 0% interest over 10 years SAVEGREEN pays NJCEP rebate when OBRP is used II 10%-19.99% $2,000 III 20%-24.99% $4,000 >25% $5,000 C&I Direct Install 70% of retrofit costs up to $125,000 Up to $53,571 OBRP 0% interest over 2 years 24

Rebate Impact Natural Gas Savings Treatment Matched Comparison Net Savings # Therms Savings % Pre Post Raw 3,168 1,028 1,099 -71** -6.9% 1,036 1,178 -143** -13.8% 72** 7.0% Degree Day 1,102 1,026 75** 6.8% 1,098 >-1 >-0.1% 76** 6.9% PRISM 2,697 1,110 1,008 101** 9.1% 1,103 1,089 14** 1.3% 88** 7.9% 25

HPwES Impact Natural Gas Savings Treatment Matched Comparison Net Savings # Therms Savings % Pre Post Raw 1,156 1,023 899 125* 12.2% 1,027 1,144 -117** -11.4% 242** 23.6% Degree Day 1,048 849 199** 19.0% 1,050 1,072 -22** -2.1% 221** 21.1% PRISM 1,068 1,039 814 225** 21.6% 1,037 -11** -1.0% 235** 22.6% **Denotes significance at the 99 percent level. 26

Environmental Benefits Methodology Energy Savings Reduction in Natural Gas Usage Avoided Emissions Tons of CO2, SO2, Nox, PM 2.5, and VOC Value Avoided Emissions $ value using APEEP Model 27

Environmental Benefits Avoided Emissions   Rebate HPwES OBRP 2013 Participants 6,700 1,720 Natural Gas Savings Per Participant (Therms) 76 221 Total Natural Gas Savings (Therms) 506,631 380,316 Total Natural Gas Savings (MMBtu) 50,663 38,032   Natural Gas Emission Rate (Tons CO2-eq/1,000 MMBtu) Marginal Value of Avoided Emissions (2015 dollars /Ton) 1 CO2-eq2 62 $41.4 SO23 0.000293 $111,573 NOx3 0.046 $23,023 PM 2.53 0.000927 $468,563 VOC3 0.00268 $44,180 28

Environmental Benefits Value of Avoided Emissions   Rebate HPwES OBRP Total Savings Avoided Emissions (tons) Marginal Value of Avoided Emissions ($ per ton) Savings CO2-eq 3,139 $41.4 $129,945 2,356 $97,547 $227,492 SO2 0.015 $111,573 $1,654 0.011 $1,242 $2,896 NOx 2.32 $23,023 $53,485 1.74 $40,150 $93,634 PM 2.5 0.047 $468,563 $22,002 0.035 $16,516 $38,518 VOC 0.136 $44,180 $6,005 0.102 $4,508 $10,513 Total $213,091 $159,963 $373,054 Monetary values are given in 2015 dollars Avoided emissions for CO2-eq are in metric tons. Avoided emissions for all other air pollutants are in short tons 29

Environmental Benefits Value of Avoided Emissions Time Period Rebate HPwES OBRP Total Benefit Participants 6,700 1,720 2014 Total $213,091 $159,963 $373,054 2014 per Participant $32 $93 Lifetime $1,909,622 $2,543,862 $4,453,491 Lifetime per Participant $285 $1,479 Monetary values in 2015 dollars. Lifetime benefits measured over 15-year measure life. Avoided emissions for CO2-eq are in metric tons. Avoided emissions for all other air pollutants are in short tons 30

Economic Benefits Theory SAVEGREEN spending replaces retail spending Charge on energy bill would have been spent on retail goods Retail spending replaces natural gas spending Energy savings from program spent on retail goods Increase in economic activity because replacement spending has Higher labor intensity Greater percentage spent in NJ 31

Economic Benefits Methodology Regional Input-Output Modeling System (RIMS) Produced by Bureau of Economic Analysis (BEA) Output Change = Expenditures * (Output Multiplier with Program – Output Multiplier Without Program) Employment Change = (1/$1,000,000) * Expenditures * (Employment Multiplier with Program – Employment Multiplier Without Program) 32

Economic Benefits Sources of Economic Impact NJNG Administrative Spending NJNG and NJCEP Incentives Customer Net Costs Customer Total Savings Estimates based on 2013 expenditures and participants’ savings. 33

Economic Benefits Output Multipliers Source of Economic Impact Output Multiplier With Program Output Multiplier Without Program Output Multiplier Increase Sector Multiplier NJNG Admin Spending   Labor: Admin, Prog Dev. Office admin 1.9212 Retail trade 1.7112 0.2100 General: Sales, Marketing Business supply 1.7661 0.0549 Labor: Rebates, Inspect, QC Prof, scientific, tech 1.9241 0.2129 NJNG & NJCEP Incentives NJNG HPwES Incentives Services to buildings/dwellings 1.7845 0.0733 NJNG Enhanced Rebates NJCEP Rebates Customer Net Costs HPwES Net Costs Rebate Net Costs Customer Total Savings Natural Gas Savings Natural gas 1.2638 0.4474 34

Economic Benefits Employment Multipliers Source of Economic Impact Employment Multiplier With Program Employ Multiplier Without Program Employ Multiplier Increase Sector Multiplier NJNG Admin Spending   Labor: Admin, Prog Dev. Office admin 14.0136 Retail trade 16.7121 -2.6985 General: Sales, Marketing Business supply 16.3152 -0.3969 Labor: Rebates, Inspect, QC Prof, scientific, tech 13.6578 -3.0543 NJNG & NJCEP Incentives NJNG HPwES Incentives Services to buildings/dwellings 22.062 5.3499 NJNG Enhanced Rebates NJCEP Rebates Customer Net Costs HPwES Net Costs Rebate Net Costs Customer Total Savings Natural Gas Savings Natural gas 2.7693 13.9428 35

Economic Benefits Output Impact Source of Economic Impact Base Amount Output Multiplier Economic Impact ($) With Program Without Program Change NJNG Admin Spending   Labor: Admin, Prog Dev. $251,346 1.9212 1.7112 0.2100 $52,783 General: Sales, Marketing $1,941,029 1.7661 0.0549 $106,562 Labor: Rebates, Inspect, QC $1,224,056 1.9241 0.2129 $260,602 NJNG & NJCEP Incentives NJNG HPwES Incentives $23,694,720 1.7845 0.0733 $1,736,823 NJNG Enhanced Rebates $4,804,500 $352,170 NJCEP Rebates $2,178,460 $159,681 Customer Net Costs HPwES Net Costs $1,568,640 $114,981 Rebate Net Costs $35,030,970 $2,567,770 Customer Total Savings Natural Gas Savings $10,086,713 1.2638 0.4474 $4,512,795 Total Economic Impact $9,864,167 36

Economic Benefits Employment Impact Source of Employment Impact Base Amount Employment Multiplier Economic Impact (Job-Years) With Program Without Program Change NJNG Admin Spending   Labor: Admin, Prog Dev. $251,346 14.0136 16.7121 -2.6985 -1 General: Sales, Marketing $1,941,029 16.3152 -0.3969 Labor: Rebates, Inspect, QC $1,224,056 13.6578 -3.0543 -4 NJNG & NJCEP Incentives NJNG HPwES Incentives $23,694,720 22.062 5.3499 127 NJNG Enhanced Rebates $4,804,500 26 NJCEP Rebates $2,178,460 12 Customer Net Costs HPwES Net Costs $1,568,640 8 Rebate Net Costs $35,030,970 187 Customer Total Savings Natural Gas Savings $10,086,713 2.7693 13.9428 141 Total Employment Impact 495 37

Economic Benefits Output and Employment Type of Impact Total Lifetime Impact Per Participant Lifetime Impact Output ($) $9,864,167 $1,172 Employment (job-years) 495 0.06 38

Health & Safety Benefits Methodology EnergySavvy Data Available for 2014 participants Issues identified during NJNG Audits for Enhanced Rebates and $6500 OBRP Percent who address issues is not known 39

Health & Safety Issues Identified   Homes With Issue Rebate $6,500 OBRP # % Improper Dryer Venting 894 23% 61 25% Improper Bath Venting 724 19% 62 Did Not Pass Oven Inspection 189 5% 6 2% Moisture Issues 132 3% 10 4% Gas Piping Leaks 90 4 Did Not Pass Depressurization Worst Case Test 76 -- Did Not Pass Water Heater Draft (Worst Case) 52 1% Gas Piping Leaks at Dryer 28 0% Did Not Pass Water Heater Draft (Natural) 15 <1% Asbestos Issues CO ≥100ppm by Water Heater 8 Structure Issues 7 CO ≥100ppm in CAZ 40

Health & Safety Benefits Participants with Any Issue   Customer Has Any Issue Rebate $6,500 OBRP # % Yes 1,616 42% 99 40% No 1,969 51% 147 60% No Health & Safety Data 283 7% 0% Total 3,868 100% 246 41

USING NEI ESTIMTES IN COST-BENEFIT ANALYSIS 42

Comparing Costs And Benefits Analysis 1 Energy Costs (exclude H&S costs) Energy Impacts Analysis 2 All Costs Analysis 3 All Impacts Need to track costs for energy and health and safety investments separately. 43

SUMMARY 44

Non-Energy Impacts Methodologies have been developed to measure NEIs Current literature on NEIs has many challenges Additional research is needed Difficult to apply findings from previous studies Factors specific to programs, jurisdictions, participants, and implementation can impact the NEIs Additional challenges relate to valuing benefits relating to health, comfort, and safety 45

Contact Jackie Berger APPRISE 32 Nassau Street, Suite 200 Princeton, NJ 08540 609-252-8009 jackie-berger@appriseinc.org www.appriseinc.org 46