1. Washington State Weatherization Program Evaluation Julie Palakovich Weatherization Program Manager DOE National Weatherization Conference December 9, 2007

2. Strategy for Washington’s 2007 Weatherization Program Evaluation: Figure out how to get blood from a turnip in less than six months.

3. Motivation for Evaluation  Respond to demands from the legislature and public for government accountability.  Last State weatherization evaluation done by ORNL in 2001.  State recognizes need for an on-going evaluation program as a performance management tool.  State and local agencies value taking a proactive approach to maintaining and improving high quality program services.

4. No easy task……  26 Local Weatherization Agencies  39 Counties  63 Utilities  4 Weatherization Programs DOE WX LIHEAP WX Bonneville Power Administration WX Energy Matchmakers Program  Measly data collected at State level

5. Evaluation Process First Step: SEEK PROFESSIONAL HELP WSU Extension Energy Program Rick Kunkle, Research Engineer Website: http://www.energy.wsu.edu

6. Next Steps……..  Convene Evaluation Working Group  Develop a WX Program Evaluation Plan  Create a mock WX Evaluation Report  Collect and analyze data  Incorporate findings into the Evaluation Report

7. Washington State Weatherization Program Evaluation Plan Section 1: Evaluation Overview o About the Program o Program Management and Funding o Past Evaluation Results o Purpose o Summary of Evaluation Methodology and Implementation

8. Evaluation Methodology and Implementation −Program Description and Delivery −Program Benefits (NOT just energy!) −Program Costs and Cost Effectiveness −Program Recommendations −Evaluation Implementation Process

9. Section 2: Evaluation Details oEvaluation Measures oData collection oAnalysis oReporting oDevelopment and implementation of the evaluation process.

10. Evaluation Measures 1. Program Description and Delivery oNumber of homes weatherized oMeasures implemented oTypes of agencies delivering WX services oMethod of service delivery oAgency accomplishments oService delivery challenges or problem areas oDemographics oHousehold survey information

11. Evaluation Measures 2. Program Benefits oHousehold oRatepayer oSocietal oEnergy

12. Evaluation Measures 3.Program Costs oAdministration oProgram Operation oHealth and Safety oWeatherization-Related Repair oTraining and Technical Assistance

13. Evaluation Measures 4.Program Cost Effectiveness -Costs and benefits for electrically heated and gas heated weatherized homes -Weatherization benefit cost ratios -Levelized cost

14. Evaluation Measures 5.Program Recommendations -Recommendations from local agencies for improving the weatherization program -Training priorities identified by local agencies -Overall recommendations for improving the weatherization program

15. Sources for Data Collection  Local Agencies  Agency Work Plan submitted to CTED  Agency Monitoring Visit Reports  Weatherization Training Survey  Household Surveys  Oak Ridge National Laboratory (Non-Energy Benefits of the WX Program)  Utility Bills and Billing Analysis  Cost-Effectiveness Analysis

16. Agency Data Collection  About 4000 units weatherized per year  10 of our 26 agencies complete 82% of the units weatherized  Developed Local Agency Data Request spreadsheet and instructions  Sent draft out to the local agencies  Held two conference calls to explain the goal of the request  Collecting data and beginning analysis

17. Analysis  Utility Bill Analysis  Other Benefits Analysis  Cost-Effectiveness Analysis

18. Reporting  Executive Summary  Introduction  Program Description and Delivery  Program Benefits  Program Costs  Program Cost-Effectiveness  Summary and Recommendations  Appendix A. Detailed Agency Data

19. Mock Evaluation Report  Purpose: Use the end result to help better define the evaluation approach.  Questions: oAre the correct evaluation measures being used? oAre the evaluation measures presented in a meaningful way? oHow can the data for each evaluation measure be obtained?

20. NOTE! Data created for illustrative purposes only!!

21. Results  Energy Savings in Electrically Heated Homes about 5-12%  Energy Savings in Gas Heated Homes about 10-25%  Cost-Benefit Ratio usually >1  Non-Energy often greater than Energy Benefits

22. Number of Low-Income Homes Weatherized

23. Percentage of Homes Receiving Measures 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% Attic Insulation Wall Insulation Floor Insulation Windows Envelope Sealing Heating System Ducts Other Measures Repair Safety Percentage of Homes Receiving Measure

24. Program and Measure Costs

25. Household Member Demographics

26. Types of Agencies Delivering Weatherization Services

27. Size of Agencies by Number of Households Weatherized in 2006

28. Energy Savings  Electrically Heated Houses  Natural Gas Heated Houses  By Size, Type and Level of Energy Use  By Weatherization Measures

29. Electricity Savings for Electrically Heated Houses Participant Normalized Savings Control Group Normalized Savings Participant Adjusted Savings Sample Size (N)200250200 Electricity Savings (kWh/yr) 29915002546 Electricity Savings (kWh/sqft) 2.60.42.2 Electricity Savings (%) 13%2%11% Electricity Cost Savings ($/yr) 179.4630152.75

30. Distribution of Energy Savings for Electrically Heated Houses

31. Natural Gas Savings for Gas Heated Houses Participant Normalized Savings Control Group Normalized Savings Participant Adjusted Savings Sample Size (N)180210180 Natural Gas Savings (kWh/yr) 23050182 Natural Gas Savings (kWh/sqft) 0.160.030.13 Natural Gas Savings (%) 27%5%21% Natural Gas Cost Savings ($/yr) 23050182

32. Distribution of Energy Savings for Natural Gas Heated Houses

33. Energy Savings by Level of Energy Use Electrically Heated House Savings ($) Natural Gas Heated House Savings ($) Low Energy Use 120100 Medium Energy Use 190200 High Energy Use 400380 Energy Savings by Size of House Electrically Heated House Savings ($) Natural Gas Heated House Savings ($) Small House13090 Medium House180210 Large House350410 Energy Savings by Type of House Electrically Heated House Savings ($) Natural Gas Heated House Savings ($) Single Family230250 Mobile Home180 Multi-Family150NA

34. Estimated Energy Cost Savings for Weatherization Measures

35. Participant Feedback  How Participants Heard About Program  Reason for Participating  Satisfaction with the Program  What Participants Learned  Actions Taken by Participants

36. How Households heard about the Weatherization Program

37. Primary Reason Households Participated in the Weatherization Program

38. Satisfaction of Households with the Weatherization Program

39. What Households Learned as a Result of Participating in the Weatherization Program

40. Actions Taken by Households as a Result of Participating in the Weatherization Program

41. Development and Implementation of Evaluation Process  Outline of potential steps for developing and implementing an on-going Weatherization Program Evaluation  Pilot evaluation proposed for first year using sample of agencies to test the waters  Based on experience, revise evaluation plan before implementing for entire Weatherization Program

42. For more information: Julie Palakovich Weatherization Program Manager CTED Housing Division PO Box 42525 Olympia, WA 98504-2525 Phone: 360-725-2948 E-mail: juliep@cted.wa.gov