Research, Evaluation, and Performance Measurement Jackie Berger Atlantic City Electric Energy Assistance Summit August 25, 2016

APPRISE Background Nonprofit Research Institute Founded in 2002 Princeton, NJ Research conducted across the U.S. Nonprofit Research Institute Low-Income Energy Bill Payment Assistance Low-Income Energy Efficiency Residential Energy Efficiency Commercial and Industrial Energy Efficiency Energy Program Research and Evaluation Federal government State government offices Utility companies Nonprofits Our Clients 2

NJ Low-Income Energy Programs Federal Block Grant Low-Income Home Energy Assistance Program (LIHEAP) Weatherization Assistance Program (WAP) State Programs NJ Universal Service Fund Program (USF) NJ Comfort Partners 3

APPRISE NJ Experience Energy Payment Assistance NJ SHARES Annual Evaluations 2006-2016 NJ Universal Service Program 2005 NJ LIHEAP and USF 2011 Low-Income Energy Efficiency NJ Comfort Partners 2002 NJ WAP 2004 NJ Comfort Partners Seniors Pilot 2005 NJ Comfort Partners 2013 Other Energy Efficiency NJ Residential New Construction Baseline 2001 NJ Energy Star Homes 2009 NJ Clean Energy Economy 2014 NJNG SAVEGREEN 2015 SJG Energy Efficiency 2016 4

Presentation Outline Research Evaluation Performance Measurement WHAT? Background Information Understand Need and Context for Programs Example: NJ Needs Assessment Evaluation Program Process and Impact Document Impacts and Assess How to Improve Program Example: BGE Pilot Payment Program Performance Measurement Program Performance Assess Opportunity for Improvement and Measure Improvement Over Time Example: NJ SHARES WHAT? WHY? HOW? 5

Evaluation and Performance Measurement Comparison Periodic In-Depth External Performance Measurement Ongoing Developmental Internal 6

Evaluation What are the program goals? How is my program performing compared to goals or expectations? How does it compare to other programs? How can the program improve? 7

Performance Measurement How can I measure? My organization’s efforts and inputs Outcomes of those efforts How we impacted clients How we impacted the utility How has this changed over time? How does my program/organization compare? What are higher performers doing? Are those designs/actions related to results? Can I implement those designs/actions? 8

research 9

American Community Survey Data Represents NJ in 2014 Number low-income under various definitions Household characteristics Energy bills Energy burden 10

NJ Program Eligibility 11

Percent Eligible 12

Number Eligible 13

Main Heating Fuel 14

Home Ownership 15

Language Spoken 16

Electric Bills Non-Electric Heaters 17

Electric Bills Electric Heaters 18

Energy Burden Gas Heaters 19

Energy Burden Electric Heaters 20

evaluation 21

Why Evaluate? “Measurement is the first step that leads to control and eventually to improvement. If you can’t measure something, you can’t understand it. If you can’t understand it, you can’t control it. If you can’t control it, you can’t improve it.” ― H. James Harrington 22

Why Evaluate Measure Program Impacts Assess Potential Improvements Meet Regulatory Requirements 23

Existing limited-income discount program BGE Pilot Motivation Incentive for on-time bill payment But only 27% receive credit for timely bill payment  Existing limited-income discount program Attempt to cost-effectively increase on-time payment Test different programs and benefits Determine impacts on payments and usage Determine cost-effectiveness New pilots 24

Pilot Programs CAMP 1-Double bill credit 2-Existing credits and payment counseling 3-Double bill credit and payment counseling GRAD 1-Graduated credits 2-Graduated credits and Quick Home Energy Check-up 3-Graduated credits and payment counseling 25

CAMP Credits Poverty Level Monthly CAMP Credit Historical Pilot ≤75% $12 $24 76% - 110% $9 $18 111% - 150% $7 $14 151% - 175% $5 $10 Subsidized Housing 26

Monthly Usage (Therms) GRAD Credits Monthly Usage (kWh) Discount or Credit Monthly Usage (Therms) ≤500 40% ≤40 501-750 30% 41-60 751-1,000 20% 61-80 1,001-1,500 10% 81-120 >1,500 $15 credit >120 $10 credit 27

CAMP Pilot Credits # Mean Credits Number Total Average Credit All CAMP 824 4.5 $51 $11 1 – Double Credits 291 $59 $13 2 – Payment Counseling 233 4.7 $32 $7 3 – Credits & Counseling 300 4.3 $58 28

GRAD Pilot Credits # Mean Credits Number Total Average Credit ALL GRAD 822 8.5 $239 $28 1 – Graduated Discount 304 8.7 $243 2 – Discount & Audit 261 8.2 $228 3 – Discount & Counseling 257 $244 29

Bill Payment Since you have been participating in this program, would you say you have been paying your BGE bill on time more often, you have been paying your BGE bill on-time less often, or there has been no change in when you pay your BGE bill? CAMP GRAD ALL 1 2 3 Double Credit Payment Counseling Credit & Counseling Discount Discount & Audit Discount & Counseling More Often 44% 48% 40% 49% Less Often 6% 7% 4% 8% 5% No Change 50% 45% 56% 46% 47% 30

Program Participation Has your participation in the program over the past year led to your participation in other energy programs such as: the Weatherization Assistance Program, BGE’s Limited Income Energy Efficiency Program, or any other energy program? CAMP GRAD ALL 1 2 3 Double Credit Payment Counseling Credit & Counseling Discount Discount & Audit Discount & Counseling WAP 24% 23% 26% 28% 21% 43% LIEEP 20% 16% 27% 31% 11% Other Energy Program 37% 19% 31

CAMP Arrearage Impacts Treatment Group Net Change # Pre Post Gross Change All CAMP 566 $197 $230 $33** -$60** 1 – Double Credits 204 $181 $202 $21 -$72** 2 – Payment Counseling 160 $187 $216 $29 -$64 3 – Credits & Counseling 202 $220 $268 $48* -$45 32

GRAD Arrearage Impacts Treatment Group Net Change # Pre Post Gross Change ALL GRAD 561 $276 $251 -$26** -$119** 1 – Graduated Discount 213 $257 $254 -$3 -$96** 2 – Discount &Audit 170 $332 $289 -$43* -$136** 3 – Discount & Counseling 178 $246 $210 -$36** -$129** 33

Recommendations Pilot Design Self-selection: those who responded to letters enrolled – difficult to extrapolate to all customers. Stratification: done differently for CAMP, GRAD, and comparison group – difficult to estimate and compare results CAMP: # on-time payments, poverty level GRAD: # on-time payments, electric usage, arrearages Enrollee tracking: difficult to examine data attrition issues. 34

Recommendations Administration Could respond to customer questions Would not have to refer to BGE Data access for DEF payment counselors Social Security recipients had trouble with payments Change bill due date to align with benefit payment at customer’s request Payment timing 35

Recommendations Customer Education Customers were unaware of program elements Many communication opportunities Invitation letter Phone enrollment Confirmation letter Shorten and simplify written communication 36

Recommendations Implementation Stratification – represent all customers to be targeted by full scale implementation Customer targeting – target those likely to have beneficial outcome Program potential – examine potential cost savings against potential costs Cost effectiveness – structure payment to be no less than what was paid prior to program 37

Performance measurement 38

Types of Measures More difficult to obtain data Inputs Outputs Outcomes Impacts Staff Hours Equipment Supplies # Applied # Enrolled % Vulnerable $ in Benefits # Referred Bill Reduction Burden Reduction % Paid Bill % Terminated More powerful information 39

NJ SHARES Input Example 250 agencies deliver grants 300 sites where clients can apply 269 events to raise awareness Individual Contributions $198,185 Corporate Contributions $461,361 Fundraising $1,993,948 40

NJ SHARES Output Example NJ SHARES serves needy households Children under the age of six: 20% Single parent households: 22% Annual income below $50,000: 58% Have family member over 60: 22% NJ SHARES serves the working poor 82% of households have employment income 6% of households receive unemployment benefits 10% of 2013 grantees received unemployment benefits 5% received unemployment from 2006-2008 (pre-recession) NJ SHARES provides grants to those in temporary need of assistance 77% received a grant in only one of the past 9 years Only 8% received a grant in more than two of the past 9 years In 90 days before grant, recipients averaged 2.2 payments and $434 in payments 41

NJ SHARES Grant Guidelines - Maximum Grant Amounts 2005 2006-2007 2008-2013 2014-2015 Electric Only $250 $300 $500 Gas Only $700 Electric & Gas $1,000 $1,200 Electric Heat Oil/Propane -- 42

Output Example % Receiving Max Grant Not updated 43

Outcome Example Grant Coverage By Grant Type Q1 and Q2 2014 Recipients Electric Only Gas Only Electric & Gas Electric Heat Number of Customers 60 41 313 Mean Pre-Grant Balance $739 $878 $1,421 $1,440 Mean Grant $429 $632 $929 $645 Mean Post-Grant Balance $310 $246 $491 $795 Mean Percent of Pre-Grant Balances Covered 78% 77% 85% 69% 44

Maximum Grant Assessment 63% of electric-only 2014 grantees received the maximum amount Compared to 84% in both 2012 and 2013 78% of pre-grant balances were covered by electric-only grants Compared to 58% in 2012 and 70% in 2013 Increase in electric-only grant amount from $300 to $500 was effective 73% of electric heat recipients received the maximum of $700 Electric heat grants cover 69% of pre-grant balances Compared to 78% for electric-only grants, 77% for gas-only grants, and 85% for electric and gas grants Consider increase in maximum electric heat grant? 45

Impact Example: Segmentation Analysis Successful (38%) Marginal Success (5%) Need More Help (57%) 46

Payment Compliance Analysis Segmentation Analysis Year After Grant Receipt Q1 2006 Q1 2007 Q1 2008 Q1 2009 Q1 2010 Q1 2011 Q1 2012 Q1 2013 Q1 2014 Q1 & Q2 2014 Successful 26% 24% 19% 32% 49% 29% 39% 38% Marginal Success 7% 6% 5% 4% Need More Help 67% 70% 76% 61% 62% 44% 69% 66% 57% TOTAL 100% 47

Impact Example Segmentation Analysis Successful (65%) Marginal Success (8%) Need More Help (27%) 48

Impact Example Segmentation Analysis Q1 2011 Recipients Q1 2012 Recipients Q1 2013 Recipients Q1 2014 Recipients Q1 & Q2 2014 Recipients Year After Grant Receipt First Year After Grant Receipt First Second Successful 49% 50% 26% 53% 29% 67% 39% 38% Marginal Success 7% 12% 5% 10% 8% 4% Need More Help 44% 37% 69% 66% 25% 57% Accounts Included 1,429 1,089 672 569 497 318 152 316 49

Impact Example Segmentation Analysis Q1 & Q2 2014 Recipients Balance Increased by $100 - $399 Balance Increased by $400 - $999 Balance Increased by $1,000 + Number of Customers 67 75 37 Percent of Customers 21% 24% 12% Mean Charges $1,873 $2,526 $4,052 Mean Payments $1,627 $1,865 $2,213 50

Impact Example Segmentation Analysis of Elderly Households Q1 & Q2 2014 Recipients Elderly Only Non-Elderly Only Difference Number of Customers 48 268 -- Percent of Customers 15% 85% Pre-Grant Balance $1,225 $1,284 -$29 Grant Amount $732 $815 -$83* Post-Grant Balance $523 $469 $54 # % Success 26 54% 94 35% 19%** Marginal Success 0% 17 6% -6%* Needs More Help 22 46% 157 59% -13% ** Statistically significant at the 95% level * Statistically significant at the 90% level 51

Data sources 52

Agency Records Most accessible Should be put in a database May not be needed if good program database Data Customers served Characteristics – income, poverty level, elderly, children Services provided 53

Public Use Data Available for free download Characterize eligible population in service territory Programming skills needed Number eligible Geography Characteristics – income, poverty level, elderly, children, language Energy costs Data 54

Customer Survey Real time feedback Requires staff time Document methodology Data Customer characteristics Satisfaction Self-reported impacts 55

Program Database Program manager – state or utility Canned reports Queries Data Customers served Characteristics – income, poverty level, elderly, children Services provided 56

Utility Data Difficult to obtain Easier for utility managed program Requires software and programming skills Customer type – heating, water heating, baseload Energy usage Energy bills Customer payments Energy assistance Collections actions Data 57

Performance Measurement Process Start with available data Identify performance measures Determine additional data sources Collect additional data Develop additional performance measures 58

Performance Measurement Repeat Compare Results Over Time Assess What is Working Refine Program 59

Performance Measurement Summary Research, evaluation, and performance measurement serve important purposes Understand program and population served Research Assess what is working and why Evaluation Measure performance over time Performance Measurement 60

President and Co-Founder Contact Jackie Berger, Ph.D. President and Co-Founder APPRISE 32 Nassau Street, Suite 200 Princeton, NJ 08542 609-252-8009 jackie-berger@appriseinc.org www.appriseinc.org   61