1. 1 Evaluation of Energy Efficiency Programs: Adjustments to Energy Savings Edward Vine California Institute for Energy and Environment Center for the Study of Energy Markets (CSEM) Policy Conference Sacramento, CA December 9, 2008

2. 2 Outline of Talk  History of Energy Efficiency in California  Overview of Evaluation  Adjustments to Energy Savings: Net Energy Savings  Cost-Effectiveness Metrics  Why do We Care about NTG & TRC?  AEA’s Guiding Principles for Evaluators

3. 3 1. History of Energy Efficiency in California  Energy efficiency is California’s highest priority resource for meeting its energy needs in a clean, reliable and low- cost manner  For more than three decades, California has adopted energy efficiency policies and made investments that are among the most aggressive in the nation  These efforts have saved more than 40,000 GWh of electricity and 12,000 MW of peak demand - avoiding the need to build 24 large (500 MW) power plants, and equal to the energy required to power 3.8 million homes

4. 4 Californians use less electricity per person than those in all other states

5. 5 California vs. US Energy Efficiency Source: California Energy Commission

6. 6 California IOU’s Investment in Energy Efficiency Source: California Energy Commission, modified by Vine (2008) Forecast Profits decoupled from sales Performance Incentives Market Restructuring Crisis IRP 2% of 2004 IOU Electric Revenues Public Goods Charges RRIM EESP Energy Action Plan

7. 7 Annual Energy Savings from EE Programs and Standards Source: Rosenfeld, California Energy Commission

8. 8 Energy Action Plan  Since its enactment in 2003, the Loading Order has been integrated into the major CPUC decisions governing energy policy and procurement.  In the EAP, the Loading Order continued:  “Pursue all cost-effective energy efficiency, first.”  Energy resources are prioritized as follows: 1.Energy efficiency/demand response 2.Renewable generation, including renewable DG 3.Increased development of affordable and reliable conventional generation 4.Transmission expansion to support all of California’s energy goals

9. 9 California: The Most Aggressive Energy Efficiency Program in the Nation  Energy Efficiency goals (2004-2013)  26,506 GWh/year  5,000 MW/year  444 Million therms/year  Eliminates need for 10 new power plants  Eliminates 9 million tons of CO 2 emissions (equal to 1.8 million cars)  $10 billion in net savings to consumers

10. 10 Current Program Cycle  2006-08  ~$2 billion in funding for 3 years  $581M in 2006, $646M in 2007, and $742 M in 2008 (excludes EM&V budget)  Annual funding from utility procurement dollars and from the Public Goods Charge  Levelized cost of 3 cents/kWh and 21 cents/therm  $2.7 billion in net savings to consumers over 3 years  $163 million EM&V budget for 3 years  8% of total portfolio funding

11. 11 Shareholder Risk/ Reward Incentive Mechanism (RRIM)  Decision 07-09-043 (Sept. 20, 2007)  Creates incentives to ensure that utility investors and managers view energy efficiency as a core part of the utility’s regulated operations that can generate meaningful earnings for its shareholders  Protects ratepayers’ financing investment  Ensure that program savings are real and verified  All calculations of the net benefits and kW, kWh and therm achievements are independently verified by the CPUC and their EM&V contractors, based on adopted EM&V protocols  Imposes penalties for substandard performance

12. 12 Adopted Incentive Mechanism Earnings/ Penalty Curve Reward (% of PEB) (per unit below CPUC goal) Penalty 65% 85%100% of CPUC goals ER = 9% ER = 12% Earnings capped at $450 million 0% 5 ¢ /kWh, $25/kW, 45 ¢ /therm below goals, or payback of negative net benefits (cost-effectiveness guarantee), whichever is greater. Penalty capped at $450 million. Earnings = ER x PEB PEB= Performance Earnings Basis ER= Earnings Rate (or Shared- Savings Rate)

13. 13 Earnings Claim & Recovery Process  Two interim claims during each 3-year program cycle  Progress payments towards total expected earnings  One final true-up claim after the program cycle is completed  Hold back: 30% of the expected earnings in each interim claim to provide a margin for error in expected earnings

14. 14

15. 15 Scope of Strategic Plan  Includes everything the two state energy agencies (CPUC and CEC) are currently working on, and more  Incorporates:  Market transformation  Voluntary market actions – to become “Business as Usual in California  Collaboration -- Roles for Local Governments, other State Agencies and Private Sector Players  4 Big Bold EE strategies as cornerstones of initial bold energy-savings/outcomes

16. 16 All new commercial construction in California will be zero net energy by 2030. Commercial New Construction All new residential construction in California will be zero net energy by 2020. Residential New Construction Heating, Ventilation, and Air Conditioning (HVAC) industry will be reshaped Residential / Small Commercial HVAC BIG BOLD Energy Efficiency Strategies Low- Income Energy Efficiency All eligible low- income homes will be energy- efficient by 2020 4

17. 17 Proposed Next Program Cycle  2009-11  ~$4 billion in funding for 3 years  IOU program applications submitted and are being reviewed by the CPUC  Decision likely next year  Bridge funding adopted by CPUC to fund some of the 2006-8 programs, but no new programs  Many key questions, including:  How do (can?) the program applications support the RRIM & EESP?  Do cost-effectiveness indicators need to be adjusted to meet RRIM & EESP needs: e.g. TRC and net-to-gross?

18. 18 California’s GHG Emissions Goals  Reduce GHG emissions to 2000 levels by 2010  Reduce GHG emissions to 1990 levels by 2020  Reduce GHG emissions to 80% below 1990 levels by 2050

19. 19 Magnitude of the Challenge 1990 Emission Baseline ~173 MMT CO 2 e Reduction 80% Reduction ~341 MMT CO 2 e

20. 20 CARB - Climate Change Scoping Plan  Climate Change Scoping Plan (Oct. 2008)  Energy efficiency strategies will provide 16% of the estimated total emission reduction (169 MMTCO 2 E) by 2020  Focus: increased incentives and more stringent building codes and appliance efficiency standards  Focus is on technological improvements, reflecting national and international studies  Bottom line: Incredible challenge & we need to do much more and urgently

21. 21 2. Overview of Evaluation  Budgets for energy efficiency program implementation and evaluation have increased over time  California:  $2 billion (2006-2008); 8% ($163M) on EM&V  Shareholders incentives and risks: $450M  $3.7 billion (2006-2008) proposed  Increased interest in evaluation results!!!  But how does evaluation relate to the program planning cycle?  And how are evaluation findings used by program managers?  How to evaluate energy efficiency programs?

22. 22 Energy Efficiency Program Planning Cycle

23. 23 Evaluation Uses by Program & Portfolio Managers  Evaluation results can be used for:  Evaluating performance (“report card”)  Accountability and demonstration of success  Improving program design and implementation  Improving engineering & behavioral assumptions  Prioritizing program and portfolio budgets  Finalizing incentive payments  Resource (strategic) planning

24. 24 Six Types of Evaluation

25. 25 Impact Evaluation Time Energy Use In the Absence Of the Program Actual Impact Installation 25

26. 26 Impact Data Needed to Collect  Monthly energy consumption  Metered or monitored energy usage  Load shape data (day, season, year)  Hours of operation for building or measures  Physical characteristics of the building and equipment (size and location)  Other physical variables: temperature, flow, weather  Building occupancy schedules and occupant data

27. 27 Data Collection and Analysis Methods for Impact Evaluation  Engineering methods  Basic statistical billing analysis  Multivariate statistical analysis  End-use metering  Short-term monitoring  Integrative methods

28. 28 3. Adjustments to Energy Savings: Net Energy Savings  Gross energy savings  Net energy savings [Additionality]  Baselines (naturally occurring … energy use/ energy savings/ market adoption, etc.)  Free riders  Program spillover

29. 29 Net Energy Savings & Additionality  International Perspective:  Joint Implementation and Clean Development Mechanism  Emission reductions must be “additional to any that would otherwise occur” [“additionality criteria”] [“net emissions”]  Determining additionality requires a baseline  Focus is on environmental additionality (GHG emissions)  US/Utility Regulatory Perspective:  Similar to net energy savings [versus gross energy savings]  Determining net savings requires a baseline  Accounts for naturally occurring conservation and free riders

30. 30 Baselines [Reference Case]  What would have happened in the absence of a particular project or program? [counter-factual question ]  Need to account, if possible, for economic growth, technological changes, prices, policy or regulatory shifts, population changes, market barriers and trends, etc.  Baselines must be credible and realistic to prevent/mitigate gaming

31. 31 Free Riders #1  Activities are undertaken by participant(s) who would have conducted the same activities if there had been no project  Savings from free riders are not “additional” to what would otherwise have occurred  Therefore, subtract the free riders’ energy savings from gross savings to get net energy savings

32. 32 Free Riders #2  Example in California:  If NTG =.8, then Free Riders =.2 (i.e., 20% of energy savings was due to energy savings from free riders)  Free rider methodology  Self reports - measurement limitations: social desirability bias, cognitive dissonance, and attribution  Comparisons with non-program areas - measurement limitations: data availability, self-selection bias, non-equivalent comparison groups Net Energy Savings : Gross Energy Savings Net to Gross = (NTG)

33. 33 UNFCCC/ CDM Indicators of Project Additionality  Regulatory Additionality  Has the project been directly or indirectly mandated by law or regulation?  Investment Additionality  Would the project have occurred under the investing party’s normal investment decision rules?  Technological Additionality  Does the project involve technology or practices that go beyond conventional practice in the corresponding industry or sector?  Barrier Removal Additionality  Does the project remove or work to overcome any information, institutional or other barriers that would persist in the reference case?

34. 34 Program Spillover  Participant spillover  Not normally looked at  What program participants did “extra” as a result of participating in program  Nonparticipant spillover  Not normally looked at  What program nonparticipants did as a result of hearing about the program  Spillover methodologies  Self reports - measurement limitations: social desirability bias, cognitive dissonance, and attribution  Comparisons with non-program areas - measurement limitations: data availability, self-selection bias, non-equivalent comparison groups  Market effects (e.g., sales) - measurement limitations: key is data availability and quality

35. 35 4. Cost-Effectiveness Metrics  If TRC >1, program is cost effective  If TRC < 1, program is not cost effective (but may still want to include as part of a portfolio of programs)  Savings are adjusted by NTG  Free riders has been the sole adjustment factor so far BENEFITS [Savings] : Utility Avoided Power Cost (and small emissions adder) Costs : Total Project Costs (All program and participant costs) Total Resource Cost = (TRC)

36. 36 TRC Concerns  Is the right avoided cost being used?  Should avoided cost be based on natural gas combined cycle gas turbine (CCGT) or renewable energy (wind, solar) power plant?  Are NTG methodological issues too overwhelming?  Should some costs be subtracted because people invest in energy efficiency for non-energy reasons?  Comfort, quiet, home value, improved air quality, health, productivity, jobs, etc.  Non-energy benefits (NEBs) are not included

37. 37 Revise TRC BENEFITS [Savings] : Utility Avoided Power Cost (New Avoided Cost Plus NEBs) Costs : Total Project Costs (All program and 20% participant costs) Total Resource Cost = (TRC)

38. 38 5. Why Do We Care about TRC & NTG? #1  Regulatory objectives & concerns  Are we giving away money wisely? (Efficiency argument)  Are we giving away money to people who do not need it? (Equity argument)  In CA: focus has been on resource acquisition (RA) programs - due to RRIM  Not on market transformation (MT) programs - EESP focus  Not on market (program-centric) - EESP focus  In CA: focus is on free riders - due to RRIM  Not on spillover (participant and non-participant) - EESP focus  Not on market effects - EESP focus

39. 39 Why Do We Care about TRC & NTG? #2  Focus varies by:  Regional objectives (RA [RRIM] vs MT [EESP])  Program maturity (new versus old programs)  Concern over reducing greenhouse gases  Gross reductions  Net reductions (cap and trade; emissions trading) - Additionality  Policy issue:  Does current use of TRC and NTG inhibit new, market transforming energy efficiency interventions?  Time to look at market effects? Non-Energy Impacts?

40. 40 IEPEC Conference  To Learn More about Evaluation and These Exciting Topics, Go to:  International Energy Program Evaluation Conference (IEPEC)  August 12-14, 2009  Portland, OR  http://www.iepec.org

41. 41 Before Carl Speaks

42. 42 American Evaluation Association: Guiding Principles for Evaluators  Systematic Inquiry  Competence  Integrity / Honesty  Respect for People  Responsibilities for General and Public Welfare http://www.eval.org/Publications/GuidingPrinciples.asp