1. Why Data Matters Building and Sustaining a Business Case NEAUC Conference June 18, 2014

2. Presentation Evaluation versus Performance Measurement Types of Performance Measures –Inputs –Outputs –Outcomes –Impacts Data Sources Process 2

3. EVALUATION VERSUS PERFORMANCE MEASUREMENT 3

4. Comparison 4 Evaluation PeriodicIn-DepthExternal Performance Measurement OngoingDevelopmentalInternal

5. Evaluation What are the goals? How is my program performing compared to goals or expectations? How does it compare to other programs? How can the program improve? 5

6. Performance Measurement How can I measure? –My organization’s efforts and inputs –Outcomes of those efforts –How we impacted clients –How we impacted utility How has this changed over time? How does my organization compare? What are higher performers doing? Are those actions related to results? Can I implement those actions? 6

7. TYPES OF PERFORMANCE MEASURES 7

8. Types of Measures 8 InputsOutputsOutcomesImpacts More powerful informationMore difficult to obtain data

9. Inputs Staff hours Equipment Travel costs Supplies 9

10. Input Example Delivery Costs 10 Agency Job Cost Year # of Jobs Mean Cost Percentiles P10P25P50P75P90 2010530$2,744$764$1,297$2,423$4,105$5,155 2011762$3,436$605$2,004$3,435$4,958$5,871 2012369$4,324$904$2,622$4,310$6,039$7,068 Total1,661$3,413$763$1,679$3,314$4,912$6,070

11. Input Example Itemized Costs 11 UGIElectric 2010201120102011 Field Support$521,520$764,932$119,334$252,962 Administration$114,529$167,622$37,482$35,671 Inspections$19,840$12,830$7,385$11,390 No Measures$4,768$4,075$1,950$1,600 Rehab Pilot$94,774$68,742$0 Conservation Pilot$0$50,000$0 TOTAL$755,161$1,068,201$166,151$301,623

12. Outputs Number of customers applied Number of customers enrolled Service delivered Participant characteristics Services coordinated with other programs 12

13. Output Example Customers Served 13 Number Referrals51,868 Defaulted10,166 Cancelled17,006 Graduates1,011 Moved8,480 Re-certifications8,512 New Enrollments19,401

14. Output Example Customer Characteristics 14 Federal Poverty Level 201120122013 NumberPercentNumberPercentNumberPercent 0-50%89225%86028%73926% 51-100%1,25236%1,07935%90632% 101-150%83224%72523%63323% Unknown53315%45014%51719% Total3,509100%3,114100%2,795100%

15. Output Example Customer Characteristics 15 201120122013 NumberPercentNumberPercentNumberPercent Child2,13362%1,92362%1,74964% Elderly36911%31710%32512%

16. Output Example Service Type 16 EnrollmentsRecertifications Phone731257 Mail14,7484,663 Office1,141299 Home10-- Total16,6305,219

17. Output Example Payment Type 17 Payment TypeEnrollments Minimum Payment11% Percent of Bill Payment54% Percent of Income Payment16% Annualized Average Payment7% Agency Selected Payment12%

18. Output Example Grant Type 18 Number of Grants PercentPaymentsPercent Electric3,35087%$783,57685% Oil38010%$106,46911% Natural Gas722%$19,4182% Propane431%$9,9111% Kerosene211%$5,9821% Coal2<1%$7380% Total3,868100%$926,094100%

19. Output Example Measures Installed 19 Measures 20112012 #%#% Audit74398%35295% LIURP Intake73997%35295% Repairs67589%31886% Blower Door Test58777%30382% Air Leakage Reduction53370%27675% Door Work52669%28076% Air Sealing49965%28276% Efficiency Testing44659%20556% HVAC Clean & Tune39552%21959% Pipe Insulation40453%21358% Energy Education34545%26271% Window Repair/Replace37850%18550% Measures 20112012 #%#% Water Measure32743%19753% Attic Insulation32042%20455% Lighting24833%14840% Mileage33043%21759% Dryer Work23130%16545% Ventilation16021%11331% Wall Insulation15921%9526% Thermostat10013%10027% Insulation9613%4312% Header Insulation14018%6818% Attic Preparation8311%4412% Refrigerator567%205%

20. Outcomes Reduction in bill Reduction in energy burden 20

21. Outcome Example Burden Reduction 21 2009 National LIHEAP Survey Pre- LIHEAP Post- LIHEAP ≤ 5%13%37% 6% - 10%32%29% 11% - 15%19%17% 16% - 20%15%8% 21% - 25%7%4% >25%14%7%

22. Outcome Example Burden Reduction 22 2009 National LIHEAP Survey AllSeniorDisabled Child Under 18 Non- Vulnerable Pre-LIHEAP16%14%17%16% Post-LIHEAP11%9%11%12%10%

23. Outcome Example Heat Restoration 23 2009 National LIHEAP Survey Restored Heat Due to Electric/ Gas Shut Off Restored Heat Due to Fuel Running Out Restored Heat Due to Broken Equipment 61%69%58%

24. Impacts Bill payment coverage rates increased Service terminations declined Energy usage declined 24

25. Impact Example Customer Survey 25 Has the winter temperature improved, worsened, or stayed the same since receiving weatherization services? Change in Winter TemperatureNumber of Respondents Improved19 Worsened1 Stayed the Same0 Did Not Notice a Difference8 Total28

26. Impact Example Arrearage Impacts 26 Pilot Payment Program Treatment Group Net Change #PrePost Gross Change ALL GRAD561$276$251-$26**-$119** 1 – Graduated Discount213$257$254-$3-$96** 2 – Discount &Audit170$332$289-$43*-$136** 3 – Discount & Counseling178$246$210-$36**-$129**

27. Impact Example Service Terminations 27 Service Terminations Utility 1Utility 2Utility 3Utility 4 Pre5%14%15%4% Post7%8%4%2% Gross Change1%-6%**-10%**-3%** Net Change-1%-6%**-10%**0%

28. Impact Example Usage Impacts 28 Obs. UsageSavingsNet Savings PrePost kWh/ ccf % % Electric Baseload 62 14,10813,4756334.5%9156.5% Electric Heating 4327,80826,3431,465 * 5.3%1,2954.7% Gas Heating 4711,5881,411177 ** 11.2%156 ** 9.8%

29. Impact Example Major Measures 29 # Major Measures Obs. Usage (ccf)Savings PrePostccf% 0581,5751,408167 ** 10.6% 11981,5551,47580 ** 5.1% 22541,5431,380162 ** 10.5% 31671,5941,368226 ** 14.2% 4 or more531,6101,340271 ** 16.8% All7301,5651,402163 ** 10.4%

30. Impact Example Cost-Effectiveness 30 ccf Saved Mean Cost Payback Years $/ccf Saved Lifetime Air Sealing/ Leakage Reduction 7.68$50913$1.25 Attic Insulation4.68$91521$2.06 Wall Insulation8.27$1,52012$1.16 Misc. Insulation6.81$1,19015$1.41 Window Work51$83416$1.58 Thermostat57$1422$0.24

31. DATA SOURCES 31

32. Agency Records Most accessible Should be put in a database May not be needed if good program database Data –Customers served –Characteristics – income, poverty level, elderly, children –Services provided 32

33. Public Use Data Available for free download Characterize eligible population in service territory Programming skills needed Data –Number eligible –Geography –Characteristics – income, poverty level, elderly, children, language –Energy costs 33

34. Customer Survey Real time feedback Requires staff time Document methodology Data –Customer characteristics –Satisfaction –Self-reported impacts 34

35. Program Database Program manager – state or utility Canned reports Queries Data –Customers served –Characteristics – income, poverty level, elderly, children –Services provided 35

36. Utility Data Difficult to obtain Easier for utility managed program Requires software and programming skills Data –Customer type – heating, water heating, baseload –Energy usage –Energy bills –Customer payments –Energy assistance 36

37. PROCESS 37

38. Process Start with available data Identify performance measures Determine additional data sources Collect additional data Develop additional performance measures 38

39. SUMMARY 39

40. Summary Performance measurement overlaps with evaluation Start with program goals Work with available data Identify ways to enhance data Measure performance over time Identify areas for improvement Impact measures require more data and analysis 40

41. Contact Jackie Berger, Ph.D. President and Co-Founder APPRISE 32 Nassau Street, Suite 200 Princeton, NJ 08542 609-252-8009 jackie-berger@appriseinc.org www.appriseinc.org 41