1. Savings Estimation Methods for Energy Efficiency Programs: A Half-Hour Guide Kansas Corporation Commission Workshop on Energy Efficiency March 25, 2008 Marian Brown Manager of Measurement & Evaluation Energy Efficiency Division Southern California Edison marian.brown@sce.com (626) 812-7662

2. Southern California Edison 2 Why Measure Energy Efficiency Program Savings? Metering the un-metered resource Measured savings for resource planning Measured benefits from reduced pollution/GHG Accountability for ratepayer dollars and shareholder reward Improve programs More accurate savings estimates for measures Increased energy savings from better design Basis for future EE program funding

3. Southern California Edison 3 Types of Programs and Impact Evaluations Resource Acquisition Programs – Create direct energy savings Education/Information Programs Create indirect energy savings Market Transformation Programs Create long-term changes in product availability, cost and features, plus customer awareness, understanding and purchase behavior

4. Southern California Edison 4 Direct Energy Savings Program participation is defined as purchasing or adopting the energy efficiency measures. Thats how rebate and direct installation programs work. So the first job is to estimate the energy savings achieved from those adoptions (called gross energy savings). Then the follow-up question is in which cases the program caused the purchase/installation/adoption (net energy savings; attribution; net-to-gross ratio).

5. Southern California Edison 5 Indirect Energy Savings The program activity is intended to influence a decision to adopt efficiency measures, rather than require it as a condition for participation. Examples: Energy audits, energy efficiency training, education, information, market transformation. So the evaluation must first determine whether the program did indeed lead to adoption of energy- efficient measures (attribution/net-to-gross ratio). Then, it must estimate what the resulting energy savings were.

6. Southern California Edison 6 Estimating Gross Energy and Demand Savings For Both Direct and Indirect Savings Programs

7. Southern California Edison 7 The Two Basic Approaches Statistical Billing (Energy Use) Analysis Simple billing analysis normalized for weather Regression analysis Multi-stage regressions (can include net savings analysis) Engineering Measurement and Analysis Pre-post measurements and engineering algorithms Pre-post building energy simulation models Sometimes, the two approaches are combined. We try to include economic/behavioral effects wherever possible.

8. Southern California Edison 8 Which Measurement & Analysis Method to Use for the Energy Savings? Methods range from simple and direct to complex and indirect and sometimes combined. More complex methods generally require more detailed data and higher cost. Guidelines for good measurement/analysis: The Model Energy Efficiency Program Impact Evaluation Guide covers both billing analysis/regression methods and engineering methods, plus net savings methods. IPMVP (International Performance Measurement & Verification Protocol) is a widely recognized standard for engineering measurement and analysis approaches.

9. Southern California Edison 9 Billing Analysis Tends To Be Preferred When: Both pre- and post-retrofit billing data are available. Expected energy savings are large enough to be statistically observable in a billing analysis (at least 10% of total usage; depends on several factors). Analysis can include large numbers of participants that are reasonably homogeneous. Inclusion of a comparison or control group allows for behavioral change corrections and/or simultaneous estimation of net savings

10. Southern California Edison 10 Engineering Measurement & Analysis Tends To Be Preferred When: No pre-installation billing data are available (e.g., new construction). Expected energy savings are a small fraction of total energy usage. Program has a small number of participants or unique measures. Program itself includes substantial engineering analysis that can be built on.

11. Southern California Edison 11 Using a Sample to Estimate Program Savings Choosing when to use a sample: Can you get good enough data for the whole population of participants? (And what is the whole population?) Or should you gather more detailed and accurate data for a good sample? Questions to ask to assure reliable results from a sample: Is it unbiased (or properly weighted to be unbiased)? Non-response bias (do some groups respond at lower rates?) Survey selection bias: Did it exclude relevant groups from the sample? Can you think of some examples? Does it give the needed level of precision? Depends on sample size and variance in the population. How far off can an estimate be and still be useful to you?

12. Southern California Edison 12 A Key Question: What Would Energy Use Be in the Absence of the Program? We can measure energy use before the program. We can measure energy use after the program. But how much of the change was due to the program ? Conventionally, we look at this question in two dimensions: One related to gross energy savings; One related to attribution and net energy savings.

13. Southern California Edison 13 Gross Energy Savings Baseline What technology/behavior choice was being made? Pure retrofit: Early replacement or add-on Then savings is prior energy use minus new energy use Replacement on burnout or new construction (lost opportunity market) Then savings is efficient equipment energy use minus usage of a standard efficiency new measure

14. Southern California Edison 14 Determining Attribution, the Net-to- Gross Ratio, and Net Energy Savings Attribution: What energy savings are the result of the program, rather than other influences?

15. Southern California Edison 15 Indirect Savings Programs Attribution is the first question for audit, training, education, marketing, and market transformation programs. Step 1: Who was actually exposed to the program? Step 2: How many of those exposed to the program activities changed their behavior in energy-saving ways? Step 3: Of those who changed, how much of the changes were due to the program rather than other influences? Step 4: Estimate the energy savings of those changes identified as due to the program.

16. Southern California Edison 16 Direct Savings Programs Attribution is usually the second question asked for rebate and direct install programs. Step 1: How much energy did participants save? Step 2: What fraction of participants changes were due to the program, rather than other causes?

17. Southern California Edison 17 The Traditional Direct Savings Approach: The Net-to-Gross Ratio = The fraction of the apparent program energy savings that are truly the result of the program Example: Program pays rebates to 100 customers who install high efficiency windows and claims the energy savings of these 100 Sources of difference for apparent vs. actual: Free riders 20 of these customers were already planning to do this without the rebate offer. Spillover 10 neighbors also install these windows because of their installing neighbors recommendations, but dont get the rebate.

18. Southern California Edison 18 Measurement Methods for the Net-to- Gross Ratio (and their Key Difficulties) Self-Report Survey Minimize bias towards self-credit for decisions Regression Analysis Comparison of Participants and Non-Participants Identify a comparable group of non-participants Market Baseline Find data for proportion of efficient sales before and after program, for the program region and a comparison region

19. Southern California Edison 19 When the Traditional Net-to-Gross Approach Doesnt Work When one or more related programs have long-term effects on a market, estimating free riders and even some short-term forms of spillover misses the major, long-term sources of savings due to the program(s). Then, we need to monitor how the market itself changes over time and try to determine how much of the change is due to the programs. The next presentation will discuss the measurement of programs with these types of market transformation effects.

20. Southern California Edison 20 Attribution Estimation is Difficult and Imprecise. Cant We Just Skip It? No! We cant just pretend a substantial effect doesnt exist, because its hard to measure. We must pay attention to how much difference the program makes, both for: Accountability to ratepayers; and Picking the right measures and the right programs to maximize cost-effectiveness.

21. Southern California Edison 21 References and Website Locations (1) Basic National Sources for Program Evaluation Methods The Model Energy Efficiency Program Impact Evaluation Guide, developed by US EPA and the National Action Plan for Energy Efficiency, http://www.epa.gov/cleanenergy/documents/evaluation_guide.pdf The International Performance Measurement & Verification Protocol, developed by the Efficiency Valuation Organization, http://www.evo-world.org/http://www.evo-world.org/ Californias database of deemed energy savings estimates for energy efficiency measures, the Database for Energy Efficiency Resources (DEER): http://eega.cpuc.ca.gov/deer/ http://eega.cpuc.ca.gov/deer/

22. Southern California Edison 22 References and Website Locations (2) Basic California Energy Efficiency Analysis Sources The California Standard Practice Manual (definitions and rules for cost-effectiveness tests) http://www.calmac.org/events/SPM_9_20_02.pdf http://www.calmac.org/events/SPM_9_20_02.pdf Californias database of deemed energy savings estimates for energy efficiency measures, the Database for Energy Efficiency Resources (DEER): http://eega.cpuc.ca.gov/deer/http://eega.cpuc.ca.gov/deer/ The California Energy Efficiency Evaluation Framework, a comprehensive and in-depth guide to all aspects of EE program evaluation: http://www.calmac.org/events/California_Evaluation_Framework_June_2004.pdf http://www.calmac.org/events/California_Evaluation_Framework_June_2004.pdf California Energy Efficiency Evaluation Protocols, the California Public Utilities Commission CPUC) rules on how to conduct EE program evaluations: http://www.calmac.org/events/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf http://www.calmac.org/events/EvaluatorsProtocols_Final_AdoptedviaRuling_06-19-2006.pdf CALMAC Website: http://www.calmac.org/. Toolkit sections has several CPUC and California EE evaluation guidance documents. Searchable Database section has over 600 California energy efficiency studies, including impact evaluations, process evaluations, market analysis studies. All of the California utilities EE evaluation studies are posted here.http://www.calmac.org/